Tracing an MLP

Forward Pass · Backward Pass · Gradient Descent · Batches & Epochs

Contents

1. Network Architecture
2. What the Weight Matrices Actually Mean
3. Dataset and Training Setup
4. The Forward Pass
5. The Backward Pass
6. Gradient Descent: Where Learning Happens
7. Batches — Aggregating Gradients Before Updating
8. Epochs — The Full Training Loop
9. Summary: Every Step and Its Role

1. Network Architecture

A small network:

• 2 inputs: two features $x_0$ and $x_1$
• 2 hidden neurons: $h_0$ and $h_1$, both using sigmoid activation $\sigma$
• 1 output neuron: $\hat{y} \in (0,1)$, the predicted probability of the positive class
• Loss: Binary cross-entropy (BCE)

1.1 The Full Network Diagram

1.2 How to Read the Diagram

Every arrow represents one number: a weight. There are exactly 6 weight arrows and 3 bias arrows. The matrices $\mathbf{W_1}$, $\mathbf{W_2}$, $\mathbf{b_1}$, $\mathbf{b_2}$ are just a compact way of writing down all those arrows.

2. What the Weight Matrices Actually Mean

2.1 W1 — Input-to-Hidden Weights

$\mathbf{W_1}$ is shape $2 \times 2$ because there are 2 hidden neurons (rows) and 2 inputs (columns). The rule is: row = destination, column = source.

$$\mathbf{W_1} = \underbrace{\begin{pmatrix} 0.5 & -0.3 \\ 0.2 & 0.8 \end{pmatrix}}_{\text{row 0 = weights arriving at }h_0,\ \ \text{row 1 = weights arriving at }h_1}$$

Sign rule: positive weight $\Rightarrow$ larger input $\Rightarrow$ neuron fires more. Negative weight $\Rightarrow$ larger input $\Rightarrow$ neuron fires less. Magnitude $\Rightarrow$ strength of the influence.

2.2 b1 — Hidden Layer Biases

$$\mathbf{b_1} = \begin{pmatrix} +0.1 \\ -0.1 \end{pmatrix}$$

The bias is an always-on input of 1 multiplied by this stored value. It shifts the neuron's firing threshold independently of any input.

• $b_1[0] = +0.1$: $h_0$ gets a small head-start. Even if $x_0 = x_1 = 0$, $h_0$'s pre-activation starts at $+0.1$ instead of zero.
• $b_1[1] = -0.1$: $h_1$ has a handicap. Inputs must overcome $-0.1$ before it fires with any conviction.

Without biases, every decision boundary in every neuron would be forced to pass through the origin — a severe restriction on what the network can represent.

2.3 W2 — Hidden-to-Output Weights

Only one output neuron, so $\mathbf{W_2}$ has only one row — effectively a flat vector:

$$\mathbf{W_2} = \begin{pmatrix} +0.7 & -0.4 \end{pmatrix}$$

$h_0$ and $h_1$ pull the output in opposite directions. The final prediction is their weighted tug-of-war.

2.4 b2 — Output Bias

$$b_2 = +0.2$$

Single scalar. Before any input is considered, the output pre-activation starts at $+0.2$, which pushes $\hat{y}$ slightly above 0.5 (towards predicting the positive class by default). Training will adjust this.

2.5 Why We Pick Any Values At All — The Symmetry Problem

We cannot start all weights at zero. If every weight is identical, every neuron in a layer produces the same output, receives the same gradient, and gets the same update — they are stuck in perfect symmetry forever and the network cannot specialise.

We initialise with small random values to break this symmetry. The specific values here are chosen for readability, not realism. In practice, schemes like Xavier or He initialisation are used. But whatever we start with, training will overwrite them.

3. Dataset and Training Setup

Batch size $= 2$ (two samples per weight update). Learning rate $\alpha = 0.1$.

4. The Forward Pass

We trace Sample 1: $x = [1.0,\ 0.5]$, true label $y = 1$.

4.1 Step 1 — Hidden Pre-Activations z1

Each hidden neuron takes a weighted sum of all inputs plus its bias. This is literally just the dot product of one row of $\mathbf{W_1}$ with the input vector, plus the bias.

$$\begin{aligned} z_1[0] &= \underbrace{W_1[0,0]}_{x_0 \to h_0} \cdot x_0 + \underbrace{W_1[0,1]}_{x_1 \to h_0} \cdot x_1 + b_1[0] \\ &= (0.5)(1.0) + (-0.3)(0.5) + 0.1 \\ &= 0.500 - 0.150 + 0.100 = \mathbf{0.45} \end{aligned}$$ $$\begin{aligned} z_1[1] &= W_1[1,0] \cdot x_0 + W_1[1,1] \cdot x_1 + b_1[1] \\ &= (0.2)(1.0) + (0.8)(0.5) + (-0.1) \\ &= 0.200 + 0.400 - 0.100 = \mathbf{0.50} \end{aligned}$$

$z_1$ is called the pre-activation: the raw weighted sum, before any squashing.

4.2 Step 2 — Hidden Activations a1

We pass each $z_1$ through the sigmoid: $\displaystyle \sigma(z) = \frac{1}{1 + e^{-z}}$

$$a_1[0] = \sigma(0.45) = \frac{1}{1 + e^{-0.45}} = \frac{1}{1 + 0.6376} \approx \mathbf{0.6106}$$ $$a_1[1] = \sigma(0.50) = \frac{1}{1 + e^{-0.50}} = \frac{1}{1 + 0.6065} \approx \mathbf{0.6225}$$

4.3 Step 3 — Output Pre-Activation z2

$$\begin{aligned} z_2 &= W_2[0] \cdot a_1[0] + W_2[1] \cdot a_1[1] + b_2 \\ &= (0.7)(0.6106) + (-0.4)(0.6225) + 0.2 \\ &= 0.4274 - 0.2490 + 0.200 = \mathbf{0.3784} \end{aligned}$$

4.4 Step 4 — Output ŷ

$$\hat{y} = \sigma(0.3784) = \frac{1}{1 + e^{-0.3784}} \approx \mathbf{0.5935}$$

The network predicts a 59.35% probability for the positive class. The true label is 1. The prediction is wrong-ish — not confident enough in the right direction.

4.5 Step 5 — Loss L

Binary cross-entropy. For $y = 1$, the second term vanishes:

$$L = -y \log \hat{y} - (1-y)\log(1 - \hat{y}) = -\log(0.5935) \approx \mathbf{0.522}$$

The loss is a single scalar measuring how bad the prediction was. The job of the backward pass is to figure out which weights caused this 0.522 and by how much.

5. The Backward Pass

The mechanism is the chain rule. The loss $L$ depends on $\hat{y}$, which depends on $z_2$, which depends on $\mathbf{W_2}$ and $\mathbf{a_1}$, which depends on $z_1$, which depends on $\mathbf{W_1}$ and $x$. We unwind this chain in reverse.

For Sample 1, the scalar computation graph is:

So the full reverse route is:

$$L \to \hat{y} \to z_2 \to \begin{cases} W_2, b_2,\\ a_1 \to z_1 \to W_1, b_1. \end{cases}$$

The output error signal is not a new assumption. It is a shortcut name for one derivative:

$$\delta_2 = \frac{\partial L}{\partial z_2}.$$

Once $\delta_2$ is known, the output-layer parameter gradients become one-line chain-rule calculations.

We define the error signal $\boldsymbol{\delta}$ at each layer:

$$\delta^{(l)} = \frac{\partial L}{\partial z^{(l)}}$$

This answers: "how urgently does this layer's pre-activation need to change?" It is the central quantity flowing backwards through the network.

5.1 Step 1 — Output Error Signal δ2

For a general binary label $y \in \{0,1\}$, the BCE loss is

$$L = -\Big(y\log(\hat{y}) + (1-y)\log(1-\hat{y})\Big).$$

Therefore

$$\frac{\partial L}{\partial \hat{y}} = -\frac{y}{\hat{y}} + \frac{1-y}{1-\hat{y}}.$$

For Sample 1, $y=1$, so this reduces to $\partial L/\partial\hat{y}=-1/\hat{y}$.

$$\delta_2 = \frac{\partial L}{\partial z_2} = \underbrace{\frac{\partial L}{\partial \hat{y}}}_{-1/\hat{y}} \cdot \underbrace{\frac{\partial \hat{y}}{\partial z_2}}_{\hat{y}(1-\hat{y})}$$

For BCE loss + sigmoid output, these factors cancel exactly:

$$\boxed{\delta_2 = \hat{y} - y = 0.5935 - 1 = \mathbf{-0.4065}}$$

If we do not use the shortcut, the Sample 1 calculation is:

$$\frac{\partial L}{\partial z_2} = \left(-\frac{1}{0.5935}\right)(0.5935)(1-0.5935) = -0.4065.$$

So yes: $\hat{y}-y$ is just a compact shortcut for the full chain $L \to \hat{y} \to z_2$.

Interpretation: the negative sign says $z_2$ must increase (the network was too low). The magnitude 0.4065 says how urgently.

This cancellation ($\hat{y} - y$) is one reason BCE + sigmoid is preferred over MSE + sigmoid. With MSE the gradient contains an extra $\sigma'(z_2)$ factor which approaches zero when the network is confidently wrong — learning stalls at exactly the worst moment.

5.2 Step 2 — Gradients for W2 and b2

Since $z_2 = W_2[0]\cdot a_1[0] + W_2[1]\cdot a_1[1] + b_2$, we have $\partial z_2 / \partial W_2[i] = a_1[i]$. By chain rule:

$$\frac{\partial L}{\partial W_2[i]} = \frac{\partial L}{\partial z_2}\frac{\partial z_2}{\partial W_2[i]} = \delta_2 a_1[i], \qquad \frac{\partial L}{\partial b_2} = \frac{\partial L}{\partial z_2}\frac{\partial z_2}{\partial b_2} = \delta_2 \cdot 1.$$ $$\frac{\partial L}{\partial W_2[0]} = \delta_2 \cdot a_1[0] = (-0.4065)(0.6106) = \mathbf{-0.2482}$$ $$\frac{\partial L}{\partial W_2[1]} = \delta_2 \cdot a_1[1] = (-0.4065)(0.6225) = \mathbf{-0.2530}$$ $$\frac{\partial L}{\partial b_2} = \delta_2 = \mathbf{-0.4065}$$

All negative: we should increase all three parameters to reduce the loss. Makes sense — the output was too low ($\hat{y} = 0.594$) for a true positive case.

5.3 Step 3 — Propagate Error Signal Back Through W2

Before we can compute gradients for $\mathbf{W_1}$, we need to know how each hidden activation $a_1[i]$ contributed to the loss. By chain rule:

$$\frac{\partial L}{\partial a_1[i]} = \frac{\partial L}{\partial z_2}\frac{\partial z_2}{\partial a_1[i]} = \delta_2 W_2[i].$$ $$\frac{\partial L}{\partial a_1[0]} = \delta_2 \cdot W_2[0] = (-0.4065)(+0.7) = \mathbf{-0.2846}$$ $$\frac{\partial L}{\partial a_1[1]} = \delta_2 \cdot W_2[1] = (-0.4065)(-0.4) = \mathbf{+0.1626}$$

$h_0$ needs to increase its output (negative gradient means increasing $a_1[0]$ decreases $L$). $h_1$ needs to decrease its output — because its weight to the output is $-0.4$, increasing $h_1$ would push $\hat{y}$ down, which makes things worse.

5.4 Step 4 — Pass Back Through the Sigmoid

To get the error signal at the hidden pre-activations $z_1$, we multiply by the sigmoid's derivative.

Recall that if $a = \sigma(z)$, then $\sigma'(z) = a(1 - a)$.

$$\delta_1[i] = \frac{\partial L}{\partial z_1[i]} = \frac{\partial L}{\partial a_1[i]}\frac{\partial a_1[i]}{\partial z_1[i]} = \frac{\partial L}{\partial a_1[i]}\sigma'(z_1[i]).$$ $$\sigma'(z_1[0]) = a_1[0]\,(1 - a_1[0]) = (0.6106)(0.3894) = 0.2378$$ $$\sigma'(z_1[1]) = a_1[1]\,(1 - a_1[1]) = (0.6225)(0.3775) = 0.2350$$ $$\delta_1[0] = \frac{\partial L}{\partial a_1[0]} \cdot \sigma'(z_1[0]) = (-0.2846)(0.2378) = \mathbf{-0.0677}$$ $$\delta_1[1] = \frac{\partial L}{\partial a_1[1]} \cdot \sigma'(z_1[1]) = (+0.1626)(0.2350) = \mathbf{+0.0382}$$

5.5 Step 5 — Gradients for W1 and b1

Finally, each hidden pre-activation is

$$z_1[i] = W_1[i,0]x_0 + W_1[i,1]x_1 + b_1[i].$$

Thus

$$\frac{\partial L}{\partial W_1[i,j]} = \frac{\partial L}{\partial z_1[i]}\frac{\partial z_1[i]}{\partial W_1[i,j]} = \delta_1[i]x_j, \qquad \frac{\partial L}{\partial b_1[i]} = \delta_1[i].$$ $$\frac{\partial L}{\partial W_1[0,0]} = \delta_1[0] \cdot x_0 = (-0.0677)(1.0) = \mathbf{-0.0677}$$ $$\frac{\partial L}{\partial W_1[0,1]} = \delta_1[0] \cdot x_1 = (-0.0677)(0.5) = \mathbf{-0.0339}$$ $$\frac{\partial L}{\partial W_1[1,0]} = \delta_1[1] \cdot x_0 = (+0.0382)(1.0) = \mathbf{+0.0382}$$ $$\frac{\partial L}{\partial W_1[1,1]} = \delta_1[1] \cdot x_1 = (+0.0382)(0.5) = \mathbf{+0.0191}$$ $$\frac{\partial L}{\partial \mathbf{b_1}} = \boldsymbol{\delta}_1 = [\mathbf{-0.0677},\ \mathbf{+0.0382}]$$

We now have a gradient for every single parameter in the network. The backward pass is complete for Sample 1.

6. Gradient Descent: Where Learning Happens

The update rule:

$$w \;\leftarrow\; w - \alpha \cdot \frac{\partial L}{\partial w}$$

Move each weight a small step opposite to its gradient:

• Negative gradient $\Rightarrow$ increasing $w$ decreases $L$ $\Rightarrow$ move $w$ up
• Positive gradient $\Rightarrow$ increasing $w$ increases $L$ $\Rightarrow$ move $w$ down

$\alpha$ is the learning rate: the step size. Too large and you overshoot and diverge. Too small and training takes forever. We use $\alpha = 0.1$.

We do not apply this update yet. In mini-batch gradient descent we first average gradients across all samples in a batch, then apply one update. This is explained in the next section.

7. Batches — Aggregating Gradients Before Updating

7.1 Forward and Backward Pass for Sample 2: x = [0.2, 0.9], y = 0

Forward pass:

$$z_1[0] = (0.5)(0.2) + (-0.3)(0.9) + 0.1 = 0.100 - 0.270 + 0.100 = -0.070$$ $$z_1[1] = (0.2)(0.2) + (0.8)(0.9) + (-0.1) = 0.040 + 0.720 - 0.100 = +0.660$$ $$a_1[0] = \sigma(-0.070) \approx 0.4825 \qquad a_1[1] = \sigma(0.660) \approx 0.6593$$ $$z_2 = (0.7)(0.4825) + (-0.4)(0.6593) + 0.2 = 0.3378 - 0.2637 + 0.200 = 0.2741$$ $$\hat{y} = \sigma(0.2741) \approx 0.5681$$ $$L = -\log(1 - 0.5681) = -\log(0.4319) \approx 0.840$$

The network predicts 56.8% positive for a true-negative sample. Both samples in Batch 1 are currently wrong.

Backward pass:

$$\delta_2 = \hat{y} - y = 0.5681 - 0 = +0.5681$$ $$\begin{aligned} \frac{\partial L}{\partial W_2[0]} &= (+0.5681)(0.4825) = +0.2741, \\ \frac{\partial L}{\partial W_2[1]} &= (+0.5681)(0.6593) = +0.3745, \\ \frac{\partial L}{\partial b_2} &= +0.5681 \end{aligned}$$ $$\frac{\partial L}{\partial a_1[0]} = (+0.5681)(0.7) = +0.3977, \quad \frac{\partial L}{\partial a_1[1]} = (+0.5681)(-0.4) = -0.2272$$ $$\sigma'(z_1[0]) = (0.4825)(0.5175) = 0.2497, \quad \sigma'(z_1[1]) = (0.6593)(0.3407) = 0.2246$$ $$\delta_1[0] = (+0.3977)(0.2497) = +0.0993, \quad \delta_1[1] = (-0.2272)(0.2246) = -0.0510$$ $$\begin{aligned} \frac{\partial L}{\partial W_1[0,0]} &= +0.0199, & \frac{\partial L}{\partial W_1[0,1]} &= +0.0894, \\ \frac{\partial L}{\partial W_1[1,0]} &= -0.0102, & \frac{\partial L}{\partial W_1[1,1]} &= -0.0459 \end{aligned}$$ $$\frac{\partial L}{\partial \mathbf{b_1}} = [+0.0993,\ -0.0510]$$

7.2 Average the Gradients Across Batch 1

Sample 1 says "push $W_2[0]$ down." Sample 2 says "push $W_2[0]$ up." They partially disagree. The batch average computes the net direction: slightly up ($+0.013$). With thousands of samples per batch this averaging cancels individual noise and gives a stable, reliable gradient estimate.

7.3 Apply Gradient Descent: One Weight Update

$$w \leftarrow w - \alpha \cdot \bar{g}, \qquad \alpha = 0.1$$

The changes are tiny. That is expected at step 1 with a cautious learning rate. Over hundreds of epochs these small, consistent nudges accumulate into weights that produce accurate predictions.

8. Epochs — The Full Training Loop

8.1 The Nested Structure

EPOCH 1
|
|-- BATCH 1  {Sample 1, Sample 2}   (starting weights)
|   |-- Forward pass: Sample 1  -->  y_hat=0.5935, L=0.522
|   |-- Forward pass: Sample 2  -->  y_hat=0.5681, L=0.840
|   |-- Backward pass: Sample 1 -->  gradient set A
|   |-- Backward pass: Sample 2 -->  gradient set B
|   |-- average(A, B)           -->  batch gradient
|   `-- Weight update           -->  weights shift (update #1)
|
`-- BATCH 2  {Sample 3, Sample 4}   (UPDATED weights from Batch 1)
    |-- Forward pass: Sample 3  -->  y_hat=?, L=?
    |-- Forward pass: Sample 4  -->  y_hat=?, L=?
    |-- Backward pass: Sample 3 -->  gradient set C
    |-- Backward pass: Sample 4 -->  gradient set D
    |-- average(C, D)           -->  batch gradient
    `-- Weight update           -->  weights shift (update #2)

EPOCH 1 DONE: 4 samples seen, weights updated twice.

EPOCH 2: repeat from scratch, using the weights epoch 1 left behind.

8.2 Critical Point: Reset Gradients Across Batches

Within a batch: gradients are averaged across samples, then one update is applied.

Across batches within an epoch: there is no accumulation. Batch 2 runs on whatever weights Batch 1 left behind. The gradients computed in Batch 1 do not directly influence Batch 2 — they influenced the weights that Batch 2 inherits. The effect is indirect, via the updated weight values.

8.3 What the Epoch Loss Number in Training Logs Means

When training prints:

Epoch 1/100 --- loss: 0.680

That 0.680 is the average of all per-sample losses collected during forward passes of that epoch:

$$L_{\text{epoch}} = \frac{1}{N} \sum_{i=1}^{N} L_i = \frac{L_1 + L_2 + L_3 + L_4}{4}$$

This is a monitoring number. It is computed from forward passes. It does not control the weight updates. Its only purpose: tell you whether the model is improving over time. If this number decreases epoch-to-epoch, training is working.

9. Summary: Every Step and Its Role

The complete training loop in pseudocode:

for epoch in 1..num_epochs:
    shuffle(dataset)
    epoch_losses = []

    for batch in chunks(dataset, size=batch_size):

        # Forward pass --- no weight change
        for sample in batch:
            y_hat = forward(sample.x, weights)
            L     = loss(y_hat, sample.y)
            epoch_losses.append(L)

        # Backward pass --- no weight change
        grads = mean([backprop(sample, weights) for sample in batch])

        # Gradient descent --- ONLY step that changes weights
        weights = weights - alpha * grads

    print("Epoch loss:", mean(epoch_losses))   # monitoring only