What a neuron is
A neuron takes inputs, does the weighted sum you already know from Linear Regression, then runs the result through an activation — a bend that lines alone cannot make.
Gradient descent still trains it. The new piece is the bend: without an activation, stacking layers would collapse back into one big straight line.
- Neuron
- One unit: inputs → weighted sum + bias → activation → output.
- Pre-activation
- The raw sum
Σ xᵢ·wᵢ + biasbefore the bend — same number linear regression would output. - Activation
- A function applied after the sum. ReLU is
max(0, x)— off below zero, pass-through above. - Why bother
- Real prices have thresholds and curves. One straight line cannot learn “premium only above 2000 sqft.”
Purpose
Why houses need a bend
Linear regression assumes price grows steadily with features. Markets often jump: nothing special below a size, then a luxury tier. A neuron with ReLU can learn “ignore below a threshold, then add.” Stacking neurons (next lesson) combines several such rules.
- in play
Dot product plus bias — Linear Regression and Gradient Descent trained exactly this piece.
Activation
ReLU on the number line
Plot pre-activation on the X axis and ReLU output on the Y axis. Flat left of zero; diagonal on the right. Step through values.
Pre-activation is −5. ReLU clips it to 0. The neuron is “off” — it contributes nothing this pass.
What happens in this step
pre = −5 ReLU(pre) = max(0, −5) = 0
| Activation | Typical use |
|---|---|
| ReLU | Hidden layers (default) |
| Sigmoid | Probabilities / gates — squashes to (0, 1) |
| Linear (none) | Regression output — final price stays unbounded |
Solution in TypeScript
preActivation is linear regression. forward adds ReLU. train is gradient descent with one extra gate: if the neuron was off (pre ≤ 0), the gradient is zero and weights do not move.
type Vector = number[];
function relu(x: number): number {
return Math.max(0, x);
}
class Neuron {
weights: Vector;
bias: number;
constructor(inputCount: number) {
// Small random start — all zeros make every neuron learn the same thing
this.weights = Array.from({ length: inputCount }, () => Math.random() * 0.2 - 0.1);
this.bias = 0;
}
/** Weighted sum before activation — same idea as linear regression. */
preActivation(inputs: Vector): number {
let sum = this.bias;
for (let i = 0; i < inputs.length; i++) {
sum += inputs[i] * this.weights[i];
}
return sum;
}
forward(inputs: Vector): number {
return relu(this.preActivation(inputs));
}
train(inputs: Vector, target: number, lr: number): void {
const pre = this.preActivation(inputs);
const output = relu(pre);
const error = target - output;
// ReLU derivative: 1 if pre > 0, else 0 (neuron "off" gets no update)
const grad = pre > 0 ? error : 0;
for (let i = 0; i < this.weights.length; i++) {
this.weights[i] += grad * inputs[i] * lr;
}
this.bias += grad * lr;
}
}
const neuron = new Neuron(3);
console.log(neuron.forward([1500, 3, 10])); // ≥ 0 after ReLU
console.log(neuron.forward([0, 0, 0])); // relu(bias) ≥ 0Limit
One neuron is not enough for XOR
Some patterns have no single straight boundary. Classic proof — XOR:
| x₁ | x₂ | want |
|---|---|---|
| 0 | 0 | 0 |
| 0 | 1 | 1 |
| 1 | 0 | 1 |
| 1 | 1 | 0 |
No single line separates the 1s from the 0s. House version: “waterfront bonus only when sqft > 2000” is an AND — one neuron cannot learn it. You need a hidden layer of several neurons.
Next
Trail so far
First Learning Model → Vectors and Weights → Linear Regression → Gradient Descent → Single Neuron → Neural Networks.