Every AI tool you used this week was trained the same way: three steps, repeated billions of times.

One step a day. Today: the forward pass.

Each node works out a weighted sum, each input times its weight, added together. Then it adds its bias and applies an activation function.

ReLU turns any negative value into zero.

The nodes in the middle are a hidden layer: their values are never in the data, only worked out on the way.

Hidden layer

Nodes between the inputs and the outputs. Their values are worked out, never given.

Weighted sum

Each value coming into a node times its connection's weight, added together.

Bias

A number added to a node's weighted sum.

Activation function

A function applied to a node's value. Here, ReLU: any negative value becomes zero.

Forward pass

Working out every node's value, layer by layer, from inputs to outputs.

Three inputs, four middle nodes, three outputs. Each road has its own path through a middle node.

1. Click on connections to find each road's path.
2. What do the 0.8 weights mean?

0.8 of green time for each car waiting.

1. Why does the network need a middle layer at all?
2. What did the activation function do in the Birthday Book Predictor?
3. If a network gives the wrong answer, which numbers would you change?

Back propagation sends the error back across each weight. A middle node receives the output's error times the weight connecting them.

A node that feeds several outputs adds up what comes back from each.

The input values never receive an error. Why not?

Error

The output value minus the target value.

Back propagation

Sending the error back across each weight, so every node knows its share of the mistake.

Middle error

The error a middle node receives: each output's error times the weight between them, added up.

Two outputs, wrappers and bottles, so each middle node collects two errors and adds them up.

Write each one as a sum of products before you type it.

The middle nodes measure the hours of sleep over and the hours short, so at most one is above zero. The other is held at zero, played no part, and receives no error.

1. Why does the error travel back across the weights?
2. Why did one yawn node receive no error?
3. What will the network do with all these errors?

Every connection now knows its share of the error. Now the network uses it to change its weights.

The learning rate keeps each step small. Moving every weight a small step against its error is gradient descent.

Weight update

Subtracting input value × error × learning rate from a weight.

Learning rate

A small number that sets how big each weight update is.

Gradient descent

Training by moving every weight a small step in the direction that makes the error smaller.

Training data

The examples a network learns from.

Testing data

Examples the network never learned from, used to check whether it really learned.

2 classes take part. The network predicts 3 bags of cans, but the target is 6. The learning rate is 0.02.

At a learning rate of 0.2, this network's weights blow up.

1. Watch it train at 0.2. What goes wrong?
2. Try the other learning rates, and record what each one does.
3. Find the one that settles.

What goes wrong above it? And below it?

The Harbor Predictor learns swimmers and sailboats from sunshine and wind, through a middle layer.

1. Train it in swift mode across all the training days.
2. Switch to the testing data, days it has never seen.
3. Which answer did it learn better?

The sailboats: off by about 0.7 a testing day, against about 1.1 for the swimmers.

1. Why multiply by the input value in the update rule?
2. What happens when the learning rate is too large? Too small?
3. Why judge the Harbor Predictor on testing data?

A weight of 4 connects an input of 3 to an output with a target of 10. The learning rate is 0.1.

1. What is the output?
2. What is the error?
3. What is the adjustment?
4. What is the new weight?

A very large set of weights and biases.

Each one found by repeating the forward pass, back propagation and the weight update until the errors are small.

Keep exploring at iinnoo.app