The Artificial Neuron · Neural Networks from First Principles

From biology to arithmetic

A biological neuron receives signals through its dendrites, adds up the excitation in its cell body, and, if the total crosses a threshold, fires a signal down its axon to other neurons.

The artificial neuron keeps only the arithmetic of that story: weigh the inputs, sum them, squash the result. It's a loose inspiration, not a model of the brain, and modern networks owe far more to mathematics than to biology.

The math of one neuron

Multiply each input by its weight, add them up with the bias, then squash: a = σ(w₁x₁ + w₂x₂ + w₃x₃ + b).

A neuron with inputs $\mathbf{x} = (x_1, \dots, x_d)$ , weights $\mathbf{w}$ and bias $b$ computes two things:

z = \sum_{j=1}^{d} w_j x_j + b = \mathbf{w}^\top \mathbf{x} + b, \qquad a = \sigma(z)

The pre-activation $z$ is a weighted sum: each weight says how much its input matters, and in which direction. The bias shifts the threshold. The activation function $\sigma$ then turns $z$ into the neuron's output $a$ .

For example, with $\mathbf{x} = (1, 2)$ , $\mathbf{w} = (0.5, -1)$ and $b = 2$ :

z = 0.5 \cdot 1 + (-1) \cdot 2 + 2 = 0.5, \qquad \mathrm{ReLU}(0.5) = 0.5, \qquad \sigma(0.5) \approx 0.62

Quick check +20 XP

A neuron has weights $\mathbf{w} = (2, -1)$ , bias $b = 0.5$ and a ReLU activation. What does it output for $\mathbf{x} = (1, 3)$ ?

The geometry: a line through the data

Here is the insight that makes neurons click. Suppose the neuron answers "class 1" when $z > 0$ . Then the points where it is exactly undecided satisfy

\mathbf{w}^\top\mathbf{x} + b = 0

In two dimensions that's the equation of a straight line. In three, it's a plane, and in general it's a hyperplane. This is the neuron's decision boundary, and its geometry is governed by the parameters:

The weight vector $\mathbf{w}$ is perpendicular to the boundary and points towards the positive side.
The bias $b$ slides the boundary back and forth without rotating it.
Scaling $\mathbf{w}$ and $b$ together doesn't move the line, but it makes a sigmoid neuron more confident: its transition gets sharper.

Try it. Rotate the line with the weights, slide it with the bias, and separate the blue points from the gold ones.

Interactive lab

One neuron, one line

A neuron computes z = w₁x₁ + w₂x₂ + b and says gold when z > 0. Its decision boundary is the line where z = 0. The arrow is the weight vector w: it always points at the gold side, perpendicular to the line.

Weight w₁1Weight w₂−0.5Bias b0.5

Activation

z = 1·x₁ − 0.5·x₂ + 0.5
a = σ(z)output at origin: 0.62

Accuracy

17/24

Activation functions

Why not skip the activation and use $z$ directly? Because stacking linear layers gets you nowhere:

W_2\big(W_1\mathbf{x} + \mathbf{b}_1\big) + \mathbf{b}_2 = \underbrace{(W_2 W_1)}_{\text{one matrix}}\mathbf{x} + \underbrace{(W_2\mathbf{b}_1 + \mathbf{b}_2)}_{\text{one bias}}

Any stack of linear layers collapses into a single linear layer. The nonlinearity is what lets depth add expressive power. Explore the classic choices, and watch their slopes, because backpropagation (next chamber) multiplies by exactly those slopes.

Interactive lab

Activation function explorer

Pick a nonlinearity, then move across the plot to read its value and its slope. The slope is what backpropagation multiplies by, so flat regions mean vanishing gradients.

Show derivative

\sigma(z) = \frac{1}{1 + e^{-z}}

\sigma'(z) = \sigma(z)\big(1 - \sigma(z)\big)

f(z)

0.731

f′(z)

0.197

Squashes any number into (0, 1), which is handy for probabilities. But it saturates: for large |z| the derivative is nearly 0, and it never exceeds 0.25.

Explored 1/6

The sigmoid has an especially neat derivative, $\sigma'(z) = \sigma(z)\big(1 - \sigma(z)\big)$ , which never exceeds $0.25$ . Hold on to that fact. It's the seed of the vanishing gradient problem.

Derive σ′(z) yourself (try it first!)

Write $\sigma(z) = (1 + e^{-z})^{-1}$ and apply the chain rule:

\sigma'(z) = \frac{e^{-z}}{(1 + e^{-z})^2} = \underbrace{\frac{1}{1 + e^{-z}}}_{\sigma(z)} \cdot \underbrace{\frac{e^{-z}}{1 + e^{-z}}}_{1 - \sigma(z)}

The second factor equals $1 - \sigma(z)$ because $\frac{e^{-z}}{1 + e^{-z}} = \frac{(1 + e^{-z}) - 1}{1 + e^{-z}}$ .

The problem that froze a field: XOR

XOR ("exclusive or") outputs 1 when exactly one of its inputs is 1:

$x_1$	$x_2$	XOR
0	0	0
0	1	1
1	0	1
1	1	0

Plot those four points and try to draw one straight line with both 1s on one side and both 0s on the other. You can't: the classes sit on opposite corners. XOR is not linearly separable, so no single neuron can compute it. Switch the lab above to XOR mode and see for yourself: 75% is the ceiling.

Key takeaways

A neuron computes a weighted sum $z = \mathbf{w}^\top\mathbf{x} + b$ , then applies an activation $a = \sigma(z)$ .
Its decision boundary $\mathbf{w}^\top\mathbf{x} + b = 0$ is a hyperplane: $\mathbf{w}$ sets the orientation, $b$ the offset.
Nonlinear activations are essential; without them, depth collapses to a single linear map.
A single neuron can only solve linearly separable problems. XOR is the classic counterexample.

Checkpoint

Prove it to the labyrinth

Answer every question to clear this chamber. First-try answers earn the most XP.

0/3

Question 1 of 3 +20 XP

Why do neural networks need nonlinear activation functions?

Question 2 of 3 +20 XP

What does the decision boundary of a single neuron look like in a 2D input space?

Question 3 of 3 +20 XP

The sigmoid's derivative is $\sigma'(z) = \sigma(z)\big(1-\sigma(z)\big)$ . What is its maximum value?

End of the chamber

Clear this chamber

Questions in this chamber (0/4 solved)Next unsolved
Bonus: Draw the line (+40 XP)
Bonus: Activation tour (+15 XP)

+50 XPArtificial NeuronActivation FunctionDecision BoundaryThe XOR Problem