Lecture 11 Part 2

Back Propagation

Abrar Hasan
Lecturer
Dept. of Software Engineering
 < T 1 >

Back Propagation
 < T 1 >

Back Propagation

Remember Gradient Descent?

Gradient descent is an optimization algorithm

commonly used to train machine learning models
and neural networks.
 < T 1 >

Chain Rule
 < T 1 >

Chain Rule
y = 5x +
3
x=
t2
 < T 1 >

Chain Rule
y = 5x +
What generally we 3
x=
do,
t2
y = 5x +
3
=
5(t2)+3
= 5t2+3

=10t
 < T 1 >

Chain Rule
y = 5x +
But in chain rule we see 3
x=
• Y is function of X t2
• X is a function of T
Instead of doing
this

(t2)
*
=5 *2t = 10t
 < T 1 >
W1

D Dimensional Input
W7 W10 y1
W2
W11
W8 W12
W3

W4 W9 W5

W5 W6
W13

Outy1= activation(w10x1+ w11x2)

< T 1 >
 < T 1 >

w1 w5
X1 H1 y1
w2 w6
w3 w7
X2 H2 y2
w4 w8
 < T 1 >
w1 w5
X1 H1 y1
w2 w6
w3 w7
X2 H2 y2
w4 w8

H1= x1*w1 + x2*w2 +b1 H2= x1*w3 + x2*w4 +b2

 < T 1 >
w1 w5
X1 H1 y1
w2 w6
w3 w7
X2 H2 y2
w4 w8

y1= out_H1*w5 + out_H2*w6 +b3 y1= outH1*w7 + outH2*w8 +b4

 < T 1 >

What we just learnt is called forward pass

Now Lets see a mathematical example

 < T 1 >

w1=.15 w5=.40 T1=.01

X1 H1 y1
.2 .4 5
= 0 =0
w2 w6
W W
3 =.2 7 =.5
5 0
X2 H2 y2
w4 = 0.3 w8 = 0.55 T2=.99
X1=.05 b1 = b2= .35
X2=.01 b3 = b4= .60
 < T 1 >
w1=.15 w5=.40
.05 H1 y1 T1=.01
.2 .45
= 0 =0
w2 w6
W W
3 =.2 7 =.5
5 0
.1 H2 y2
w4 = 0.3 w8 = 0.55
T2=.99

H1= x1*w1 + x2*w2 +b1

H1= .05*.15 + .1*.2 +.35
= .5932
= .3775
 < T 1 >
w1=.15 w5=.40 T1=.01
.05 H1 y1
.2 .45
= 0 =0
w2 w6
W W
3 =.2 7 =.5
5 0
.1 H2 y2
w4 = 0.3 w8 = 0.55
T2=.99

H2= .3925
output_H2= .59688
 < T 1 >
w1=.15 w5=.40 T1=.01
.05 H1 y1
.2 .45
= 0 =0
w2 w6
W W
3 =.2 7 =.5
5 0
.1 H2 y2
w4 = 0.3 w8 = 0.55
T2=.99

y1= out_H1*w5 + out_H2*w6 +b3

= .5932 *.4 + .59688 *.45 +.6

= .7513
= 1.1056
 < T 1 >
w1=.15 w5=.40 T1=.01
.05 H1 y1
.2 .45
= 0 =0
w2 w6
W W
3 =.2 7 =.5
5 0
.1 H2 y2
w4 = 0.3 w8 = 0.55
T2=.99

y2 = 1.22 output_y1 = .77

< T 1 >
< T 1 >

Derivative of Sigmoid
 < T 1 >

= out_H1+ 0+0
= out_H1
 < T 1 >

-(T1-outy1) out_H1
< T 1 >
 < T 1 >

Do the same for all weight

 < T 1 >

w1=.15 w5=.40 T1=.01

X1 H1 y1
=0
.2 0.45
=
w2 w6
W W
3 =.2 7 =.5
5 0
X2 H2 y2
w4 = 0.3 w8 = 0.55 T2=.99
 < T 1 >

w1=.15 w5=.40 T1=.01

X1 H1 y1
=0
.2 0.45
=
w2 w6
W W
3 =.2 7 =.5
5 0
X2 H2 y2
w4 = 0.3 w8 = 0.55 T2=.99
 < T 1 >

w1=.15 w5=.40 T1=.01

X1 H1 y1
=0
.2 0.45
=
w2 w6
W W
3 =.2 7 =.5
5 0
X2 H2 y2
w4 = 0.3 w8 = 0.55 T2=.99
 < T 1 >

w1=.15 w5=.40 T1=.01

X1 H1 y1
=0
.2 0.45
=
w2 w6
W W
3 =.2 7 =.5
5 0
X2 H2 y2
w4 = 0.3 w8 = 0.55 T2=.99
Thank
You