26/09/2023, 13:07 Backpropagation Example With Numbers Step by Step – A Not So Random Walk

Menu

A Not So Random
Walk
About

A Not So Random Walk

Backpropagation Example With Numbers

Step by Step
Posted on February 28, 2019

When I come across a new mathematical concept or before I use a canned software
package, I like to replicate the calculations in order to get a deeper understanding of
what is going on. This type of computation based approach from first principles
helped me greatly when I first came across material on artificial neural networks.

In this post, I go through a detailed example of one iteration of the backpropagation

algorithm using full formulas from basic principles and actual values. The neural
network I use has three input neurons, one hidden layer with two neurons, and an
output layer with two neurons.

The following are the (very) high level steps that I will take in this post. Details on each
step will follow after.

(1) Initialize weights for the parameters we want to train

(2) Forward propagate through the network to get the output values

(3) Define the error or cost function and its first derivatives

(4) Backpropagate through the network to determine the error derivatives

(5) Update the parameter estimates using the error derivative and the current value

https://www.anotsorandomwalk.com/backpropagation-example-with-numbers-step-by-step/ 1/13
26/09/2023, 13:07 Backpropagation Example With Numbers Step by Step – A Not So Random Walk

The input and target values for this problem are and
. I will initialize weights as shown in the diagram below. Generally,
you will assign them randomly but for illustration purposes, I’ve chosen these
numbers.

Mathematically, we have the following relationships between nodes in the networks.

For the input and output layer, I will use the somewhat strange convention of
denoting , , , and to denote the value before the activation function is
applied and the notation of , , , and to denote the values after application of
the activation function.

Input to hidden layer

https://www.anotsorandomwalk.com/backpropagation-example-with-numbers-step-by-step/ 2/13
26/09/2023, 13:07 Backpropagation Example With Numbers Step by Step – A Not So Random Walk

Hidden layer to output layer

We can use the formulas above to forward propagate through the network. I’ve
shown up to four decimal places below but maintained all decimals in actual
calculations.

We now define the sum of squares error using the target values and the results from
the last layer from forward propagation.

We are now ready to backpropagate through the network to compute all the error
derivatives with respect to the parameters. Note that although there will be many
long formulas, we are not doing anything fancy here. We are just using the basic
principles of calculus such as the chain rule.

First we go over some derivatives we will need in this step. The derivative of the
sigmoid function is given here. Also, given that and

https://www.anotsorandomwalk.com/backpropagation-example-with-numbers-step-by-step/ 3/13
26/09/2023, 13:07 Backpropagation Example With Numbers Step by Step – A Not So Random Walk

, we have , , , ,
, and .

We are now ready to calculate , , , and using the derivatives we have

already discussed.

I will omit the details on the next three computations since they are very similar to the
one above. Feel free to leave a comment if you are unable to replicate the numbers
below.

The error derivative of is a little bit more involved since changes to affect the
error through both and .

https://www.anotsorandomwalk.com/backpropagation-example-with-numbers-step-by-step/ 4/13
26/09/2023, 13:07 Backpropagation Example With Numbers Step by Step – A Not So Random Walk

To summarize, we have computed numerical values for the error derivatives with
respect to , , , , and . We will now backpropagate one layer to compute
the error derivatives of the parameters connecting the input layer to the hidden layer.
These error derivatives are , , , , , , and .

I will calculate , , and first since they all flow through the node.

The calculation of the first term on the right hand side of the equation above is a bit
more involved than previous calculations since affects the error through both
and .

Now I will proceed with the numerical values for the error derivatives above. These
derivatives have already been calculated above or are similar in style to those
calculated above. If anything is unclear, please leave a comment.

Plugging the above into the formula for , we get

The calculations for and are below

https://www.anotsorandomwalk.com/backpropagation-example-with-numbers-step-by-step/ 5/13
26/09/2023, 13:07 Backpropagation Example With Numbers Step by Step – A Not So Random Walk

I will now calculate , , and since they all flow through the node.

The calculation of the first term on the right hand side of the equation above is a bit
more involved since affects the error through both and .

Plugging the above into the formula for , we get

The calculations for and are below

The final error derivative we have to calculate is , which is done next

https://www.anotsorandomwalk.com/backpropagation-example-with-numbers-step-by-step/ 6/13
26/09/2023, 13:07 Backpropagation Example With Numbers Step by Step – A Not So Random Walk

We now have all the error derivatives and we’re ready to make the parameter
updates after the first iteration of backpropagation. We will use the learning rate of

So what do we do now? We repeat that over and over many times until the error goes
down and the parameter estimates stabilize or converge to some values. We
obviously won’t be going through all these calculations manually. I’ve provided Python
code below that codifies the calculations above. Nowadays, we wouldn’t do any of
these manually but rather use a machine learning package that is already readily
available.

https://www.anotsorandomwalk.com/backpropagation-example-with-numbers-step-by-step/ 7/13
26/09/2023, 13:07 Backpropagation Example With Numbers Step by Step – A Not So Random Walk

https://www.anotsorandomwalk.com/backpropagation-example-with-numbers-step-by-step/ 8/13
26/09/2023, 13:07 Backpropagation Example With Numbers Step by Step – A Not So Random Walk

https://www.anotsorandomwalk.com/backpropagation-example-with-numbers-step-by-step/ 9/13
26/09/2023, 13:07 Backpropagation Example With Numbers Step by Step – A Not So Random Walk

I ran 10,000 iterations and we see below that sum of squares error has dropped
significantly after the first thousand or so iterations.

9 thoughts on “Backpropagation Example With

Numbers Step by Step”

jpowersbaseball says:
December 30, 2019 at 5:28 pm

When I use gradient checking to evaluate this algorithm, I get some odd results.
For instance, w5’s gradient calculated above is 0.0099. But when I calculate the
costs of the network when I adjust w5 by 0.0001 and -0.0001, I get 3.5365879
and 3.5365727 whose difference divided by 0.0002 is 0.07614, 7 times greater
than the calculated gradient. I think I’m doing my checking correctly?
Reply

Sebastian says:

https://www.anotsorandomwalk.com/backpropagation-example-with-numbers-step-by-step/ 10/13