Training Your First Network

From one neuron to a network

A layer is a group of neurons that all read the same inputs. Feed one layer's outputs into the next and you have a multilayer perceptron (MLP). The layers in the middle are called hidden because we never see their outputs directly.

For binary classification with one hidden layer:

Each hidden unit learns a feature, a new way of looking at the input. The output neuron then draws its straight line in feature space instead of input space. That's the whole trick: the hidden layer transforms the problem until it becomes linearly separable.

Solving XOR by hand

Here is a classic construction (it appears in Goodfellow, Bengio and Courville's Deep Learning, §6.1) that solves XOR with two ReLU hidden units:

Look at what the hidden layer did. It mapped $(0,1)$ and $(1,0)$ to the same hidden point, $(h_1, h_2) = (1, 0)$. In hidden space the problem is no longer XOR at all: a straight line finishes the job.

The training loop

Every neural network you'll ever train, from this toy to a large language model, runs the same four-beat loop:

In PyTorch it reads almost exactly like the diagram:

import torch
from torch import nn

# X: float tensor of shape (n, 2); y: float tensor of shape (n, 1) with 0/1 labels
model = nn.Sequential(nn.Linear(2, 4), nn.Tanh(), nn.Linear(4, 1))
loss_fn = nn.BCEWithLogitsLoss()     # sigmoid + cross-entropy, numerically stable
opt = torch.optim.SGD(model.parameters(), lr=0.3)

for epoch in range(1000):
    logits = model(X)                # 1. forward pass
    loss = loss_fn(logits, y)        # 2. loss
    opt.zero_grad()
    loss.backward()                  # 3. backward pass (Chamber 4)
    opt.step()                       # 4. update (Chamber 2)

Cross-entropy: the classification loss

Squared error works for regression, but for probabilities there's a better choice. If the network predicts $\hat{p}_i$ for the probability that example $i$ has label $y_i = 1$, the binary cross-entropy is

• Confident and right costs almost nothing: $-\log(0.99) \approx 0.01$.
• Confident and wrong is brutal: $-\log(0.01) \approx 4.6$.
• It's the negative log-likelihood of the data under the model, the same "loss as likelihood" idea from Chamber 1's research lens.

And it pairs beautifully with a sigmoid output. The gradient with respect to the output neuron's pre-activation $z$ is simply $\hat{p} - y$: prediction minus truth.

Train it yourself

This is a real neural network, trained live in your browser with the backprop you derived in the last chamber. Colour shows the network's confidence. Watch the boundary bend.

Things to try:

1. XOR with 1 hidden unit. However long you train, it stays stuck: one hidden unit can only draw one line.
2. XOR with 2 hidden units. Surprise: on this data it almost always fails. The hand-built solution above handles four corner points, but here the points fill whole quadrants, and two units struggle to wall off two opposite quadrants. Try 3 and 4 units, with several seeds each.
3. Circle. Watch the boundary curl into a loop.
4. Spiral. The legendary challenge. Two hidden layers, 8 units, tanh, learning rate 0.3, and some patience.
5. Break it. Push the learning rate to 3 with ReLU and watch training fall apart.

Generalisation: the real exam

A network with enough units can drive its training loss towards zero, even on random labels. So a low training loss proves little. What matters is performance on data the model has never seen, and measuring that honestly requires discipline:

• Training set: used to fit the weights.
• Validation set: used to make decisions such as learning rate, architecture and when to stop.
• Test set: touched once, at the very end, to estimate performance on truly unseen data.

Overfitting shows up as a widening gap: training loss keeps falling while validation loss turns upwards. The model is memorising noise. Underfitting is the opposite: both losses stay high because the model is too simple or undertrained.

Standard remedies for overfitting:

• More data: often the most reliable fix.
• Early stopping: keep the checkpoint with the best validation loss.
• Weight decay (L2 regularisation), which penalises large weights: $\mathcal{L}_{\text{total}} = \mathcal{L} + \lambda \lVert\theta\rVert_2^2$.
• Dropout, which randomly silences units during training so no single unit can be relied on too much.