Machine Learning - MT 2016

11 & 12. Neural Networks

Varun Kanade

University of Oxford
November 14 & 16, 2016
Announcements

I Problem Sheet 3 due this Friday by noon

I Practical 2 this week: Compare NBC & LR

I (Optional) Reading a paper

1
Outline

Today, we’ll study feedforward neural networks

I Multi-layer perceptrons

I Classification or regression settings

I Backpropagation to compute gradients

I Brief introduction to tensorflow and MNIST

2
Artificial Neuron : Logistic Regression

1 Unit

x1 w1 Σ yb = Pr(y = 1 | x, w, b)

w2

x2
Linear Function
Non-linearity

I A unit in a neural network computes a linear function of its input and is

then composed with a non-linear activation function
I For logistic regression, the non-linear activation function is the sigmoid
1
σ(z) =
1 + e−z
I The separating surface is linear
3
Multilayer Perceptron (MLP) : Classification

1
b21
2
Σ
w11
x1 3
w11
2
w12
2
1 b31 Σ yb = Pr(y = 1 | x, W, b)
w21 3
x2 w12
2
w22
Σ
b22
1

4
Multilayer Perceptron (MLP) : Regression

1
b21
2
Σ
w11
x1 3
w11
2
w12
2
1 b31 Σ yb = E[y | x, W, b]
w21 3
x2 w12
2
w22
Σ
b22
1

5
A Toy Example

6
Logistic Regression Fails Badly

7
Solve using MLP

z12 a21
1
b21
Σ
2
w11 z13 a31
x1 3
w11
2
w12
1 b31 Σ yb = Pr(y = 1 | x, Wi , bi )
2
w21 z22 a22 3
x2 w12
2
w22
Σ
b22
1

Let us use the notation:

a1 = z1 = x
z2 = W2 a1 + b2
a2 = tanh(z2 )
z3 = W3 a2 + b3
y = a3 = σ(z3 )

8
Scatterplot Comparison (x1 , x2 ) vs (a21 , a22 )

9
Decision Boundary of the Neural Net

10
Feedforward Neural Networks
Layer 1 Layer 2 Layer 3 Layer 4
(Input) (Hidden) (Hidden) (Output)

Fully
Connected
Layer

11
Computing Gradients on Toy Example

b21 Want the derivatives

2 z12 → a21
w11
x1 3
w11
∂`
2 , ∂`
2
2 ∂w11 ∂w12
w12
∂` ∂`
b31 z13 → a31 `(y, a31 ) 2
∂w21
, 2
∂w22
2
w21 3 ∂` ∂`
x2 w12 3
∂w11
, 3
∂w12
2
w22 ∂` ∂` ∂`
z22 → a22 ∂b2
, ∂b2
, ∂b3
b22 1 2 1

∂` ∂` ∂`
Would suffice to compute 3
∂z1
, 2
∂z1
, 2
∂z2

12
Computing Gradients on Toy Example
Let us compute the following:
a31 −y
1. ∂`
∂a3
= − ay3 + 1−y
1−a3
= a3 3
1 1 1 1 (1−a1 )

3
2. ∂a
3
∂z1
= a31 · (1 − a31 )

3
∂ z1 3 3
3. ∂a2
= [w11 , w12 ]
" #
∂ a2 1 − tanh2 (z12 ) 0
4. =
∂z2 0 1 − tanh2 (z22 )

Then we can calculate

∂ a3
∂`
3
∂z1
= ∂`
∂a3
· ∂z1
1
3 = a31 − y
1
 
∂` ∂` ∂ a3 ∂ z3 ∂ a2 ∂`
3
∂ z1 ∂ a2
∂z2
= ∂a3
· ∂z3 · ∂a21 ·
1
∂z2
= 3
∂z1
· ∂a2
· ∂z2
1 1

13
 loss ` aL

layer L Each layer consists of a linear function

and non-linear activation
∂`
∂zL
Layer l consists of the following:
layer L − 1
zl = Wl al−1 + bl
al = fl (zl )
layer l where fl is the non-linear activation in
∂` layer l.
∂zl

layer l − 1
If there are nl units in layer l, then Wl is
nl × nl−1
layer 2
Backward pass to compute derivatives

∂`
input x a1 ∂z2

14
 loss ` aL

layer L Forward Equations

(1) a1 = x (input)
layer L − 1
(2) zl = Wl al−1 + bl

layer l (3) al = fl (zl )

(4) `(aL , y)
layer l − 1

layer 2

input x a1

15
Output Layer

aL
zL = WL aL−1 + bL
aL = fL (zL )
layer L (zL → aL )
Loss: `(y, aL )
∂` ∂` ∂` ∂ aL
aL−1 ∂zL ∂zL
= ∂aL
· ∂zL

∂` ∂`
If there are nL (output) units in layer L, then ∂aL
and ∂zL
are row vectors
L
∂a
with nL elements and ∂zL
is the nL × nL Jacobian matrix:
 L 
∂ a1 ∂ aL ∂ aL
1
· · · ∂zL1
 ∂z1L ∂z2 L
nL 
 ∂ aL ∂ aL ∂ aL

 L2 2
· · · ∂zL2 
∂ aL  ∂z1 ∂z2 L
nL

= . . .

∂zL  .. .. .. .. 

 L . 
 ∂ an ∂ aLnL ∂ aLnL

∂z L
L
∂z L
· · · ∂z L
1 2 nL

If fL is applied element-wise, e.g., sigmoid then this matrix is diagonal

16
Back Propagation

al (the inputs into layer l + 1)

z l+1
=W l+1 l
a +b l+1 l+1
(wj,k weight on connection from kth
l ∂`
a ∂zl+1 unit in layer l to j th unit in layer l + 1)
al = f (zl ) (f is a non-linearity)
l l
layer l (z → a ) ∂`
∂zl+1
(derivative passed from layer above)
l+1
∂` ∂` ∂z
∂` = ·
al−1 ∂zl ∂zl ∂zl+1 ∂zl
∂` ∂ zl+1 ∂ al
= ∂zl+1
· ∂al
· ∂zl
∂ al
= ∂`
∂zl+1
· Wl+1 · ∂zl

17
Gradients with respect to parameters

∂`
al ∂zl+1

zl = Wl al−1 + bl (wj,k
l
weight on connection from kth
unit in layer l-1 to j th unit in layer l)
l l
layer l (z → a )
∂`
∂zl
(obtained using backpropagation)
l−1 ∂`
a ∂zl

∂ zil
Consider ∂`
l
∂wij
= ∂`
∂zil
· ∂wijl = ∂`
∂zil
· al−1
j

∂` ∂`
∂bli
= ∂zil
 T
More succinctly, we may write: ∂`
∂Wl
= al−1 ∂z
∂`
l

∂` ∂`
∂bl
= ∂zl

18
 loss ` aL Forward Equations

(1) a1 = x (input)

layer L (2) zl = Wl al−1 + bl

∂`
∂zL (3) al = fl (zl )
layer L − 1
(4) `(aL , y)

layer l
∂` Back-propagation Equations
∂zl

layer l − 1 ∂` ∂` ∂ aL
(1) Compute ∂zL
= ∂aL
· ∂zL

∂ al
(2) ∂`
∂zl
= ∂`
∂zl+1
· Wl+1 · ∂zl
layer 2
 T
(3) ∂`
∂Wl
= al−1 ∂z
∂`
l

∂` ∂` ∂`
input x a1 ∂z2
(4) ∂bl
= ∂zl

19
Computational Questions

What is the running time to compute the gradient for a single data point?
I As many matrix multiplications as there are fully connected layers
I Performed twice during forward and backward pass

What is the space requirement?

I Need to store vectors al , zl , and ∂`
∂zl
for each layer

Can we process multiple examples together?

I Yes, if we minibatch, we perform tensor operations
I Make sure that all parameters fit in GPU memory

20
Training Deep Neural Networks

I Back-propagation gives gradient

I Stochastic gradient descent is the method of choice

I Regularisation
I How do we add `1 or `2 regularisation?
I Don’t regularise bias terms

I How about convergence?

I What did we learn in the last 10 years, that we didn’t know in the 80s?

21
Training Feedforward Deep Networks
Layer 1 Layer 2 Layer 3 Layer 4
(Input) (Hidden) (Hidden) (Output)

Why do we get non-convex optimisation problem?

All units in a layer are symmetric, hence invariant to permutations
22
A toy example

1
z12 a21
b21
Σ a21 Target is y = 1−x
2
w12
x ∈ {−1, 1}

Squared Loss Function

`(a21 , y) = (a21 − y)2
1

∂ a2
∂`
2
∂z1
= 2(a21 − y) · ∂z1
1
2 = 2(a21 − y)σ 0 (z12 ) 0.8

0.6
If x = −1, w12 ≈ 5, b21 ≈ 0, then σ 0 (z12 ) ≈ 0
0.4

Cross-Entropy Loss Function 0.2

z12
`(a21 , y) = −(y log a21 + (1 − y) log(1 − a21 )) 0

−8 −6 −4 −2 0 2 4 6 8
a21 −y ∂ a2
∂`
2
∂z1
= a2 2 · 1
2 = (a21 − y)
1 (1−a1 ) ∂z1

23
Propagating Gradients Backwards

1 1 1
b21 b31 b41
x = a11 w12 Σ w13 Σ w14 Σ a41

I Cross entropy loss: `(a41 , y) = −(y log a41 + (1 − y) log(1 − a41 ))

I ∂`
4
∂z1
= a41 − y
4
∂ z1 ∂ a3
I ∂`
3
∂z1
= ∂`
4
∂z1
· ∂a3
· ∂z1
1
3 = (a41 − y) · w14 · σ 0 (z13 )
1
3
∂ z1 ∂ a2
I ∂`
2
∂z1
= ∂`
3
∂z1
· ∂a2
· ∂z1
1
3 = (a41 − y) · w14 · σ 0 (z13 ) · w13 · σ 0 (z12 )
1

I Saturation: When the output of an artificial neuron is in the ‘flat’ part,

e.g., where σ 0 (z) ≈ 0 for sigmoid
I Vanishing Gradient Problem: Multiplying several σ 0 (zil ) together makes
the gradient ≈ 0, when we have a large number of layers
I For example, when using sigmoid activation, σ 0 (z) ∈ [0, 1/4]

24
Avoiding Saturation

Rectifier
Use rectified linear units 3

Rectifier non-linearity
f (z) = max(0, z)
2

Rectified Linear Unit (ReLU)

max(0, a · w + b)
1
You can also use f (z) = |z|

Other variants
0
leaky ReLUs, parametric ReLUs −3 −2 −1 0 1 2 3

25
Initialising Weights and Biases

Initialising is important when minimising

non-convex functions. We may get very different
results depending on where we start the
optimisation.

Suppose we were using a sigmoid unit, how would

you initialise the weights?
PD
I Suppose z =
i=1 wi ai

I E.g., choose wi ∈ [− √1D , √1 ]

D
at random

What if it were a ReLU unit?

I You can initialise similarly

How about the biases?

I For sigmoid, can use 0 or a random value
around 0
I For ReLU, should use a small positive constant

26
Avoiding Overfitting

Deep Neural Networks have a lot of parameters

I Fully connected layers with n1 , n2 , .., nL units have at least

n1 n2 + n2 n3 + · · · + nL−1 nL parameters

I For Problem Sheet 4, you will be asked to train an MLP for digit
recognition with 2 million parameters and only 60,000 training images

I For image detection, one of the most famous models, the neural net
used by Krizhevsky, Sutskever, Hinton (2012) has 60 million parameters
and 1.2 million training images

I How do we prevent deep neural networks from overfitting?

27
Early Stopping

Maintain validation set and stop training

when error on validation set stops
decreasing.

What are the computational costs?

I Need to compute validation error
I Can do this every few iterations to
reduce overhead

What are the advantages?

I If validation error flattens, or starts
increasing can stop optimisation
I Prevents overfitting

See paper by Hardt, Recht and Singer (2015)

28
Add Data: Modified Data
Typically, getting additional data is either impossible or expensive

Fake the data!

Images can be translated slight, rotated slightly, change of brightness, etc.

Google Offline Translate trained on entirely fake data!

Google Research Blog

29
Add Data: Adversarial Training

Take trained (or partially trained model)

Create examples by modifications ‘‘imperceptible to the human eye’’, but

where the model fails

Szegedy et al. and Goodfellow et al.

30
Other Ideas to Reduce Overfitting

Hard constraints on weights

Gradient Clipping

Inject noise into the system

Enforce sparsity in the neural network

Unsupervised Pre-training
(Bengio et al.)

31
Bagging (Bootstrap Aggregation)

Bagging (Leo Breiman - 1994)

I Given dataset D = h(xi , yi )iN

i=1 , sample D1 , D2 , · · · , Dk of size N from
D with replacement

I Train classifiers f1 , . . . , fk on D1 , . . . , Dk

I When predicting use majority (or average if using regression)

I Clearly this approach is not practical for deep networks

32
Dropout

I For input x each hidden unit with probability 1/2 independently

I Every input, will have a potentially different mask
I Potentially exponentially different models, but have ‘‘same weights’’
I After training whole network is used by halving all the weights

Srivastava, Hinton, Krizhevsky, 2014

33
Errors Made by MLP for Digit Recognition

34
Avoiding Overfitting

I Use parameter sharing a.k.a weight tying in the model

I Exploit invariances to translation, rotation, etc.

I Exploit locality in images, audio, text, etc.

I Convolutional Neural Networks (convnets)

35
Convolutional Neural Networks (convnets)

(Fukushima, LeCun, Hinton 1980s)

36
Image Convolution

Source: L. W. Kheng

37
Convolution

In general, a convolution filter f is a tensor of dimension Wf × Hf × Fl ,

where Fl is the number of channels in the previous layer

Strides in x and y directions dictate which convolutions are computed to

obtain the next layer

Zero-padding can be used if required to adjust layer sizes and boundaries

Typically, a convolution layer will have a large number of filters, the

number of channels in the next layer will be the same as the number of
filters used

38
Source: Krizhevsky, Sutskever, Hinton (2012)

39
Sources: Krizhevsky, Sutskever, Hinton (2012); Wikipedia

40
Source: Krizhevsky, Sutskever, Hinton (2012)

41
Source: Zeiler and Fergus (2013)

42
Source: Zeiler and Fergus (2013)
43
Convolutional Layer

Suppose that there is no zero padding and strides in both directions are 1

Wf 0 Hf 0 F
l
XXX 0
zil+1
0 ,j 0 ,f 0 = bf 0 + ali0 +i−1,j 0 +j−1,f wi,j,f
l+1,f

i=1 j=1 f =1

∂ z l+1
i ,j ,f 0
0 0
l+1,f 0
= ali0 +i−1,j 0 +j−1,f
∂wi,j,f
X
∂`
l+1,f 0
= ∂`
· ali0 +i−1,j 0 +j−1,f
∂wi,j,f ∂z l+1
i ,j ,f 0
0 0
i0 ,j 0

44
Convolutional Layer

Suppose that there is no zero padding and strides in both directions are 1

Wf 0 Hf 0 F
l
XXX 0
zil+1
0 ,j 0 ,f 0 = bf 0 + ali0 +i−1,j 0 +j−1,f wi,j,f
l+1,f

i=1 j=1 f =1

∂ z l+1 0
i ,j ,f 0
0 0 l+1,f
∂ali,j,f
= wi−i 0 +1,j−j 0 +1,f

l+1,f 0
X
∂` ∂`
∂ali,j,f
= l+1 · wi−i 0 +1,j−j 0 +1,f
∂z 0 0 0
i ,j ,f
i0 ,j 0 ,f 0

45
Max-Pooling Layer

Let Ω(i0 , j 0 ) be the set of (i, j) pairs in the previous layer that are involved in
the maxpool

sl+1
i0 ,j 0 = max ali,j
i,j∈Ω(i0 ,j 0 )
!
∂ sl+1
0 0
i ,j
∂ali,j
= I (i, j) = argmax alĩ,j̃
ĩ,j̃∈Ω(i0 ,j 0 )

46
Next Week

I Practial will be about training neural networks on MNIST dataset

I Time permitting, implement one problem on the sheet in tensorflow

I Start Unsupervised Learning

I Revise eigenvectors, eigenvalues (Problem 4 on Sheet 3)

47