Terms used in deep learning
These words are the ones from the Overview, listed so you can look one up without re-reading the whole story. Each gets a definition, the same house example, and the smallest TypeScript that shows the idea.
Weight
A dial that multiplies one input. Training is allowed to change it.
The house: A weight of 100 on size means every extra square foot adds 100 to the mix before any bend.
const sqftWeight = 100;
const fromSize = 1500 * sqftWeight; // 150000Bias
An extra dial added after the multiplies. It is not tied to one feature — it shifts the whole mix up or down.
The house: A bias of 50000 means the mix starts at 50000 before size, bedrooms, or age are counted.
const bias = 50_000;
const mix = 1500 * 100 + 3 * 20_000 + 10 * -500 + bias; // 255000Pre-activation
The mix result before the bend: weighted sum of inputs plus bias. Still a straight-line combination of the inputs.
The house: For [1500, 3, 10] with the toy dials above, the pre-activation is 255000.
function mix(inputs: number[], weights: number[], bias: number): number {
let sum = bias;
for (let i = 0; i < inputs.length; i++) sum += inputs[i] * weights[i];
return sum;
}
const pre = mix([1500, 3, 10], [100, 20_000, -500], 50_000); // 255000Activation function
A small rule applied to the pre-activation to bend it. Without a bend, stacking mixes stays linear. Hidden layers almost always use one; a dollar-amount output often uses none.
The house: After the mix computes 255000, an activation decides what number the neuron actually sends onward.
type Activation = (x: number) => number;
const identity: Activation = (x) => x; // no bend — common on price output
const relu: Activation = (x) => Math.max(0, x);ReLU
Rectified Linear Unit: max(0, x). Negative values become 0 (neuron off); positive values pass through (neuron on). The default activation in modern hidden layers.
The house: If a neuron’s mix for this house is −3, ReLU returns 0 and that neuron stays quiet for this example.
function relu(x: number): number {
return Math.max(0, x);
}
relu(255_000); // 255000
relu(-3); // 0Neuron
One small calculator: mix inputs with weights and bias, then (usually) apply an activation, and emit a single number.
The house: One neuron reads [1500, 3, 10], mixes to 255000, applies ReLU, and outputs 255000.
function neuron(inputs: number[], weights: number[], bias: number): number {
const pre = mix(inputs, weights, bias);
return relu(pre);
}
neuron([1500, 3, 10], [100, 20_000, -500], 50_000); // 255000Layer
Several neurons side by side that all see the same inputs and each produce one number. The layer’s output is that list of numbers.
The house: Two neurons on the same house might emit [255000, 0] — one on, one off.
function layer(
inputs: number[],
neurons: { weights: number[]; bias: number }[],
): number[] {
return neurons.map((n) => neuron(inputs, n.weights, n.bias));
}Hidden layer
A middle layer whose outputs are not the final answer. It reshapes the inputs into internal numbers for later layers.
The house: You care about the predicted price, not the hidden vector [12.4, 0] — but that vector is what the output neuron reads.
const features = [1500, 3, 10];
const hidden = layer(features, hiddenNeurons); // e.g. [12.4, 0]
const price = mix(hidden, outputWeights, outputBias);Neural network
Layers hooked in a line: each layer’s output list becomes the next layer’s input list.
The house: [sqft, beds, age] → hidden ReLU layer → linear output → predicted price.
function predictPrice(features: number[]): number {
const hidden = layer(features, hiddenNeurons); // ReLU inside
return mix(hidden, outputWeights, outputBias); // no ReLU
}Forward pass
One trip from inputs to prediction. Weights do not change during this trip — it is only a guess.
The house: Features go in; a dollar amount comes out. The real sale price is not used yet.
const prediction = predictPrice([1500, 3, 10]);
// weights unchanged so farBackpropagation
After measuring error at the output, walk the network in reverse so every weight gets its share of the blame, then take a gradient-descent step.
The house: Prediction was 288000, sale was 295000. Error flows to the output weights first, then into the hidden weights — skipping neurons that were ReLU-off.
const error = actualPrice - prediction;
// 1) nudge output weights using error
// 2) chain error into hidden neurons × ReLU gate (0 if off)
// 3) nudge hidden weightsDeep / deep learning
Deep means more than one nonlinear stage (usually two or more hidden layers with activations). Deep learning is machine learning that uses those deep networks.
The house: inputs → ReLU layer → ReLU layer → price is deep; inputs → one ReLU layer → price is a shallower net.
const h1 = layer(features, layer1); // ReLU
const h2 = layer(h1, layer2); // ReLU again — depth
const price = mix(h2, wOut, bOut);Learned features / representation
The internal numbers a hidden layer invents while training. You did not name them by hand; training shaped them because they helped the final prediction.
The house: A hidden vector like [12.4, 0, 3.1] is not sqft or bedrooms — it is a new encoding of the house that later layers use.
const representation = layer(features, hiddenNeurons);
// training moves hiddenNeurons so this vector becomes usefulTraining
The loop where labels are available and weights move: forward pass, measure error, backpropagate, nudge, repeat across epochs.
The house: Five houses with known sale prices. Run many epochs until predicted prices sit closer to the real ones.
for (let epoch = 0; epoch < 5_000; epoch++) {
for (const [features, price] of HOUSES) {
// forward → error → backprop → update weights
}
}Inference
Using a finished network on new data. Predict only — no label required, no weights moving.
The house: A new 2000 sqft listing arrives. The network returns a price estimate with frozen dials.
const estimate = predictPrice([2000, 3, 5]);
// weights stay frozenOverfitting
Train performance looks great while new examples do poorly — the network memorized the homework instead of a pattern that transfers. Deeper nets have more capacity, so this risk rises.
The house: Loss on four training houses is nearly zero, but the held-out fifth house is far off.
const train = HOUSES.slice(0, 4);
const heldOut = HOUSES[4];
// fit only on train, then score heldOut