Forward, Backward, Update

Introduction for students

Ask the class which AI tools they have used this week, and how they think those tools got so good. Every one was trained the same way: three steps, repeated billions of times. Today the class will do all three by hand.

Write the three steps on the board. The forward pass pushes the inputs through the network to make a prediction. Back propagation measures the error and sends it back through the network. The weight update nudges every weight to make the next error smaller.

Explain that each node works out a weighted sum, each input times its weight, added together. The nodes in the middle are a hidden layer: their values are never shown in the data, only worked out on the way.

Key Vocabulary

Hidden layer

A layer of nodes between the inputs and the outputs. Its values are worked out, never given.

Weighted sum

Each value coming into a node times its connection's weight, added together.

Forward pass

Working out every node's value, layer by layer, from the inputs to the outputs.

Error

The output value minus the target value.

Back propagation

Sending the error back across each weight, so every node knows its share of the mistake.

Learning rate

A small number that sets how big each weight update is.

Gradient descent

Training by moving every weight a small step in the direction that makes the error smaller.

Guiding the activity

Forward through the middle

Pairs open The Talent Show Predictor, which asks them to predict each middle node and the output in the first round. With 2 acts and 3 posters, the top middle node is 2 × 1 + 3 × 2 = 8 and the bottom one 2 × 2 + 3 × 1 = 7. The output is 8 × 5 + 7 × 10 = 110.

Send the error back

Next, The Street Musician Predictor asks students to predict the error that reaches the middle node. In round 1 the network predicts $20 against a target of $23, so the output error is -3. It travels back across the 0.5 weight, so the middle node receives -3 × 0.5 = -1.5. The input values receive nothing, because nothing can change them.

Update a weight

Then The Recycling Drive Predictor asks for the output error, both middle errors and one weight's adjustment in round 1. The first drive, 2 classes, predicts 3 bags against a target of 6: an error of -3, which reaches the middle nodes as -3 and -1.5. The adjustment to the top middle node's weight is input × error × learning rate, 2 × -3 × 0.02 = -0.12. Subtracting it raises the weight from 1 to 1.12.

For early finishers

The Test Score Predictor 02 learns at a rate of 0.2, and its weights blow up. Pairs try the other learning rates until they find the one that settles, 0.02, then run it again.

• Predict

  The Talent Show Predictor

  This network predicts the audience at a school talent show. The input values are the number of acts and posters put up. Each input value reaches both middle nodes, and both middle nodes reach the output value. In the first round, predict each middle node value, then the output value. Enter your answer when asked.

  Open lesson →
• Predict

  The Street Musician Predictor

  This network predicts what a street musician collects. The input values are the hours played in the morning and in the evening. The output value is the money in the hat. The input values never receive an error, because nothing can change them. Predict the error that reaches the middle node, where it stops.

  Open lesson →
• Predict

  The Recycling Drive Predictor

  This network predicts the bags of cans a school recycling drive collects. The input value is the number of classes taking part. The output error comes back to both middle nodes. In round 1, predict the output error, both middle errors, and the adjustment to the weight to the top middle node. The learning rate is 0.02.

  Open lesson →
• Edit

  The Test Score Predictor 02

  This network predicts a test score. The input values are the hours of study and of sleep. The output value is the score. Its weights blow up as it learns. Find a learning rate that keeps them under control. Figure out which learning rate works, then run it again.

  Open lesson →

Post-activity discussion

1. Why does the error travel back across the weights, rather than straight to every node?

   Listen for A weight decides how much a middle node contributed to the output, so it decides how much of the error that node is responsible for.

2. Why is the adjustment subtracted?

   Listen for A negative error means the output was too low. Subtracting a negative adjustment raises the weight, which raises the output. The update always moves against the error.

3. What went wrong with a learning rate of 0.2?

   Listen for Each step was so big that it overshot the best weight, further each time, until the weights blew up.

4. Large AI models have billions of weights. What changes, and what stays the same?

   Listen for The scale changes. The three steps, and the arithmetic in each, stay exactly the same.