Introduction to RNNs!

Arun Mallya!

Best viewed with Computer Modern fonts installed!

 Outline!
•  Why Recurrent Neural Networks (RNNs)?!
•  The Vanilla RNN unit!
•  The RNN forward pass!
•  Backpropagation refresher!
•  The RNN backward pass!
•  Issues with the Vanilla RNN!
•  The Long Short-Term Memory (LSTM) unit!
•  The LSTM Forward & Backward pass!
•  LSTM variants and tips!
–  Peephole LSTM!
–  GRU!
 Motivation!
•  Not all problems can be converted into one with fixed-
length inputs and outputs!
!
•  Problems such as Speech Recognition or Time-series
Prediction require a system to store and use context
information!
–  Simple case: Output YES if the number of 1s is even, else NO!
1000010101 – YES, 100011 – NO, … !
!
•  Hard/Impossible to choose a fixed context window!
–  There can always be a new sample longer than anything seen!
 Recurrent Neural Networks (RNNs)!
•  Recurrent Neural Networks take the previous output or
hidden states as inputs. !
The composite input at time t has some historical
information about the happenings at time T < t!

•  RNNs are useful as their intermediate values (state) can

store information about past inputs for a time that is not
fixed a priori!
!
Sample Feed-forward Network!

y1!

h1!

x1!
t = 1!

5  
 Sample RNN!

y3!

y2!
h3!
y1!
h2!
x3!
h1! t = 3!
x2!

x1! t = 2!

t = 1!

6  
 Sample RNN!

y3!

y2!
h3!
y1!
h2!
x3!
h1! t = 3!
x2!
h0! t = 2!
x1!
t = 1!

7  
The Vanilla RNN Cell!

xt! W!
!
ht!
!
ht-1!
!

⎛ xt ⎞
ht = tanhW ⎜ ⎟
⎝ ht−1 ⎠

8  
 The Vanilla RNN Forward!
C1! C2! C3!

y1! y2! y3!

⎛ xt ⎞
ht = tanhW ⎜ ⎟
⎝ ht−1 ⎠
h1! h2! h3!
yt = F(ht )
Ct = Loss(yt ,GTt )

x1 h0! x2 h1! x3 h2!

! ! !

9  
 The Vanilla RNN Forward!
C1! C2! C3!

y1! y2! y3!

⎛ xt ⎞
ht = tanhW ⎜ ⎟
⎝ ht−1 ⎠
h1! h2! h3!
yt = F(ht )
Ct = Loss(yt ,GTt )

indicates shared weights!

x1 h0! x2 h1! x3 h2!
! ! !

10  
 Recurrent Neural Networks (RNNs)!
•  Note that the weights are shared over time!

•  Essentially, copies of the RNN cell are made over time

(unrolling/unfolding), with different inputs at different
time steps!
 Sentiment Classification!
•  Classify a !
restaurant review from Yelp! OR!
movie review from IMDB OR!
…!
as positive or negative!
!
•  Inputs: Multiple words, one or more sentences!
•  Outputs: Positive / Negative classification!

•  “The food was really good”!

•  “The chicken crossed the road because it was uncooked”!
 Sentiment Classification!

RNN!
h1!

The!
 Sentiment Classification!

RNN! RNN!
h1! h2!

The! food!
 Sentiment Classification!

hn!

RNN! RNN! RNN!

h1! h2! hn-1!

The! food! good!

 Sentiment Classification!

Linear
Classifier!

hn!

RNN! RNN! RNN!

h1! h2! hn-1!

The! food! good!

 Sentiment Classification!

Linear
Ignore! Ignore!
Classifier!

h1! h2! hn!

RNN! RNN! RNN!

h1! h2! hn-1!

The! food! good!

 Sentiment Classification!

h = Sum(…)!

h1! hn!
h2!

RNN! RNN! RNN!

h1! h2! hn-1!

The! food! good!

http://deeplearning.net/tutorial/lstm.html !
 Sentiment Classification!

Linear
Classifier!

h = Sum(…)!

h1! hn!
h2!

RNN! RNN! RNN!

h1! h2! hn-1!

The! food! good!

http://deeplearning.net/tutorial/lstm.html !
 Image Captioning!
•  Given an image, produce a sentence describing its contents!
!
•  Inputs: Image feature (from a CNN)!
•  Outputs: Multiple words (let’s consider one sentence)!

: The dog is hiding  

 Image Captioning!

RNN!

CNN!
 Image Captioning!
The!

Linear
Classifier!
h2!
RNN! RNN!
h1! h2!

CNN!
 Image Captioning!
The! dog!

Linear Linear
Classifier! Classifier!
h2! h3!
RNN! RNN! RNN!
h1! h2! h3!

CNN!
 RNN Outputs: Image Captions!

Show and Tell: A Neural Image Caption Generator, CVPR 15!

 RNN Outputs: Language Modeling!

VIOLA:! KING LEAR:!

Why, Salisbury must find his flesh and thought! O, if you were a feeble sight, the
That which I am not aps, not a man and in fire,! courtesy of your law,!
To show the reining of the raven and the wars! Your sight and several breath, will
To grace my hand reproach within, and not a fair are wear the gods!
hand,! With his heads, and my hands are
That Caesar and my goodly father's world;! wonder'd at the deeds,!
When I was heaven of presence and our fleets,! So drop upon your lordship's head,
We spare with hours, but cut thy council I am great,! and your opinion!
Murdered and by thy master's ready there! Shall be against your honour.!
My power to give thee but so much as hell:!
Some service in the noble bondman here,!
Would show him to her wine.!

http://karpathy.github.io/2015/05/21/rnn-effectiveness/ !
 Input – Output Scenarios!

Single - Single! Feed-forward Network!

Single - Multiple! Image Captioning!

Multiple - Single! Sentiment Classification!

Multiple - Multiple! Translation!

Image Captioning!
 Input – Output Scenarios!
Note: We might deliberately choose to frame our problem as a!
particular input-output scenario for ease of training or!
better performance. !
For example, at each time step, provide previous word as!
input for image captioning!
(Single-Multiple to Multiple-Multiple).!
 The Vanilla RNN Forward!
C1! C2! C3!

y1! y2! y3! ⎛ xt ⎞

ht = tanhW ⎜ ⎟
⎝ ht−1 ⎠
yt = F(ht )
h1! h2! h3! Ct = Loss(yt ,GTt )

“Unfold” network through time by

making copies at each time-step!

x1 h0! x2 h1! x3 h2!

! ! !

28  
BackPropagation Refresher!

y = f (x;W )
C! C = Loss(y, yGT )

y!
SGD Update
f(x; W)! ∂C
W ←W −η
∂W
x!
∂C ⎛ ∂C ⎞ ⎛ ∂y ⎞
=⎜ ⎜ ⎟
∂W ⎝ ∂y ⎟⎠ ⎝ ∂W ⎠
 Multiple Layers!
y1 = f1 (x;W1 )
y2 = f2 (y1;W2 )
C! C = Loss(y2 , yGT )

y2!
SGD Update
∂C
f2(y1; W2)! W2 ← W2 − η
∂W2
y1! ∂C
W1 ← W1 − η
∂W1
f1(x; W1)!

x!
Chain Rule for Gradient Computation!
y1 = f1 (x;W1 )
y2 = f2 (y1;W2 )
C! C = Loss(y2 , yGT )
∂C ∂C
y2! Find ,
∂W1 ∂W2
f2(y1; W2)! ∂C ⎛ ∂C ⎞ ⎛ ∂y2 ⎞
=⎜
∂W2 ⎝ ∂y2 ⎟⎠ ⎜⎝ ∂W2 ⎟⎠
y1!
∂C ⎛ ∂C ⎞ ⎛ ∂y1 ⎞
=⎜
f1(x; W1)! ∂W1 ⎝ ∂y1 ⎟⎠ ⎜⎝ ∂W1 ⎟⎠
⎛ ∂C ⎞ ⎛ ∂y2 ⎞ ⎛ ∂y1 ⎞
x! =⎜
⎝ ∂y2 ⎟⎠ ⎜⎝ ∂y1 ⎟⎠ ⎜⎝ ∂W1 ⎟⎠
Application of the Chain Rule!
Chain Rule for Gradient Computation!
⎛ ∂C ⎞
Given:! ⎜
⎝ ∂y ⎟⎠
⎛ ∂C ⎞ ⎛ ∂C ⎞
We are interested in computing:! ⎜
⎝ ∂W ⎠ ⎝ ∂x ⎟⎠
⎟
y!
,⎜

Intrinsic to the layer are:!

f(x; W)!
⎛ ∂y ⎞
⎜⎝ ⎟⎠ − How does output change due to params
x! ∂W
⎛ ∂y ⎞
⎜⎝ ⎟⎠ − How does output change due to inputs
∂x

⎛ ∂C ⎞ ⎛ ∂C ⎞ ⎛ ∂y ⎞ ⎛ ∂C ⎞ ⎛ ∂C ⎞ ⎛ ∂y ⎞
⎜⎝ ⎟⎠ = ⎜ ⎜ ⎟ ⎜⎝ ⎟⎠ = ⎜ ⎜ ⎟
∂W ⎝ ∂y ⎟⎠ ⎝ ∂W ⎠ ∂x ⎝ ∂y ⎟⎠ ⎝ ∂x ⎠
 Chain Rule for Gradient Computation!
⎛ ∂C ⎞
Given:! ⎜
⎝ ∂y ⎟⎠
⎛ ∂C ⎞ ⎛ ∂C ⎞ ⎛ ∂C ⎞
⎜⎝ ∂y ⎟⎠ We are interested in computing:! ⎜
⎝ ∂W ⎠ ⎝ ∂x ⎟⎠
⎟ ,⎜

Intrinsic to the layer are:!

f(x; W)!
⎛ ∂y ⎞
⎜⎝ ⎟⎠ − How does output change due to params
∂W
⎛ ∂C ⎞
⎜⎝ ⎟
∂x ⎠ ⎛ ∂y ⎞
⎜⎝ ⎟⎠ − How does output change due to inputs
∂x

⎛ ∂C ⎞ ⎛ ∂C ⎞ ⎛ ∂y ⎞ ⎛ ∂C ⎞ ⎛ ∂C ⎞ ⎛ ∂y ⎞
⎜⎝ ⎟⎠ = ⎜ ⎜ ⎟ ⎜⎝ ⎟⎠ = ⎜ ⎜ ⎟
∂W ⎝ ∂y ⎟⎠ ⎝ ∂W ⎠ ∂x ⎝ ∂y ⎟⎠ ⎝ ∂x ⎠
Equations for common layers: http://arunmallya.github.io/writeups/nn/backprop.html!
Extension to Computational Graphs!

y1! y2!

y!
f1(y; W1)! f2(y; W2)!
f(x; W)!

y! y!
x!

f(x; W)!

x!
Extension to Computational Graphs!
⎛ ∂C1 ⎞ ⎛ ∂C2 ⎞
⎛ ∂C ⎞ ⎜⎝ ∂y ⎟⎠ ⎜⎝ ∂y ⎟⎠
1 2
⎜⎝ ∂y ⎟⎠
f1(y; W1)! f2(y; W2)!
f(x; W)!
⎛ ∂C1 ⎞ ⎛ ∂C2 ⎞
⎜⎝ ∂y ⎟⎠ ⎜⎝ ∂y ⎟⎠
⎛ ∂C ⎞
⎜⎝ ⎟
∂x ⎠ Σ
f(x; W)!

⎛ ∂C ⎞
⎜⎝ ⎟
∂x ⎠
Extension to Computational Graphs!
⎛ ∂C1 ⎞ ⎛ ∂C2 ⎞
⎛ ∂C ⎞ ⎜⎝ ∂y ⎟⎠ ⎜⎝ ∂y ⎟⎠
1 2
⎜⎝ ∂y ⎟⎠
f1(y; W1)! f2(y; W2)!
f(x; W)!
⎛ ∂C1 ⎞ ⎛ ∂C2 ⎞
⎜⎝ ∂y ⎟⎠ ⎜⎝ ∂y ⎟⎠
⎛ ∂C ⎞
⎜⎝ ⎟
∂x ⎠ Σ Gradient Accumulation!

f(x; W)!

⎛ ∂C ⎞
⎜⎝ ⎟
∂x ⎠
 BackPropagation Through Time
(BPTT)!
•  One of the methods used to train RNNs!
•  The unfolded network (used during forward pass) is
treated as one big feed-forward network!
•  This unfolded network accepts the whole time series as
input!

•  The weight updates are computed for each copy in the

unfolded network, then summed (or averaged) and then
applied to the RNN weights!
 The Unfolded Vanilla RNN!
C1! C2! C3!

y1! y2! y3!

•  Treat the unfolded network as one

big feed-forward network!!

h3! •  This big network takes in entire

h2! sequence as an input!
h1!
•  Compute gradients through the
usual backpropagation!
!
•  Update shared weights!
h2!
!
h1!
!
h0!
!
x1 x2 x3 38  
! ! !
The Unfolded Vanilla RNN Forward!
C1! C2! C3!

y1! y2! y3!

h1! h2! h3!

x1 h0! x2 h1! x3 h2!

! ! !

39  
The Unfolded Vanilla RNN Backward!
C1! C2! C3!

y1! y2! y3!

h1! h2! h3!

x1 h0! x2 h1! x3 h2!

! ! !

40  
 The Vanilla RNN Backward!
⎛ xt ⎞
C1! C2! C3! ht = tanhW ⎜ ⎟
⎝ ht−1 ⎠
y1! y2! y3!
yt = F(ht )
Ct = Loss(yt ,GTt )

h1! h2! h3!

∂Ct ⎛ ∂Ct ⎞ ⎛ ∂yt ⎞

=⎜
∂h1 ⎝ ∂yt ⎟⎠ ⎜⎝ ∂h1 ⎟⎠
⎛ ∂Ct ⎞ ⎛ ∂yt ⎞ ⎛ ∂ht ⎞ ⎛ ∂h2 ⎞
=⎜ !⎜
⎝ ∂yt ⎠ ⎝ ∂ht ⎠ ⎝ ∂ht−1 ⎠ ⎝ ∂h1 ⎟⎠
⎟ ⎜ ⎟ ⎜ ⎟
x1 h0! x2 h1! x3 h2!
! ! !

41  
 Issues with the Vanilla RNNs!
•  In the same way a product of k real numbers can shrink to
zero or explode to infinity, so can a product of matrices!

•  It is sufficient for λ1 < 1 / γ , where λ1 is the largest singular

value of W, for the vanishing gradients problem to occur
and it is necessary for exploding gradients that λ1 > 1 / γ ,
where γ = 1 for the tanh non-linearity and γ = 1 / 4 for the
sigmoid non-linearity 1!

•  Exploding gradients are often controlled with gradient

element-wise or norm clipping!

1 On the difficulty of training recurrent neural networks, Pascanu et al., 2013!

 The Identity Relationship!
•  Recall ! ∂Ct = ⎛ ∂Ct ⎞ ⎛ ∂yt ⎞ ⎛ xt ⎞
ht = tanhW ⎜ ⎟
∂h1 ⎜⎝ ∂yt ⎟⎠ ⎜⎝ ∂h1 ⎟⎠ ⎝ ht−1 ⎠
⎛ ∂Ct ⎞ ⎛ ∂yt ⎞ ⎛ ∂ht ⎞ ⎛ ∂h2 ⎞ yt = F(ht )
=⎜ !⎜
⎝ ∂yt ⎠ ⎝ ∂ht ⎠ ⎝ ∂ht−1 ⎠ ⎝ ∂h1 ⎟⎠
⎟ ⎜ ⎟ ⎜ ⎟ Ct = Loss(yt ,GTt )

•  Suppose that instead of a matrix multiplication, we had an

identity relationship between the hidden states!
ht = ht−1 + F(xt )
⎛ ∂ht ⎞
⇒⎜ ⎟ =1
⎝ ∂ht −1 ⎠
•  The gradient does not decay as the error is propagated all
the way back aka “Constant Error Flow”!
 The Identity Relationship!
•  Recall ! ∂Ct = ⎛ ∂Ct ⎞ ⎛ ∂yt ⎞ ⎛ xt ⎞
ht = tanhW ⎜ ⎟
∂h1 ⎜⎝ ∂yt ⎟⎠ ⎜⎝ ∂h1 ⎟⎠ ⎝ ht−1 ⎠
⎛ ∂Ct ⎞ ⎛ ∂yt ⎞ ⎛ ∂ht ⎞ ⎛ ∂h2 ⎞ yt = F(ht )
=⎜ !⎜
⎝ ∂yt ⎠ ⎝ ∂ht ⎠ ⎝ ∂ht−1 ⎠ ⎝ ∂h1 ⎟⎠
⎟ ⎜ ⎟ ⎜ ⎟ Ct = Loss(yt ,GTt )

•  Suppose that instead of a matrix multiplication, we had an

identity relationship between the hidden states!
ht = ht−1 + F(xt ) Remember Resnets?!
⎛ ∂ht ⎞
⇒⎜ ⎟ =1
⎝ ∂ht −1 ⎠
•  The gradient does not decay as the error is propagated all
the way back aka “Constant Error Flow”!
 Disclaimer!
•  The explanations in the previous few slides are handwavy!

•  For rigorous proofs and derivations, please refer to !

On the difficulty of training recurrent neural networks, Pascanu et al., 2013!
Long Short-Term Memory, Hochreiter et al., 1997!
And other sources!
!
 Long Short-Term Memory (LSTM)1!
•  The LSTM uses this idea of “Constant Error Flow” for
RNNs to create a “Constant Error Carousel” (CEC) which
ensures that gradients don’t decay!

•  The key component is a memory cell that acts like an

accumulator (contains the identity relationship) over time!

•  Instead of computing new state as a matrix product with

the old state, it rather computes the difference between
them. Expressivity is the same, but gradients are better
behaved!
!

1 Long Short-Term Memory, Hochreiter et al., 1997! 46  

!
 The LSTM Idea!

xt! W! Cell!
!
!
ct! ht!
ht-1!
!

⎛ xt ⎞
ct = ct−1 + tanhW ⎜ ⎟ ht = tanh ct
⎝ ht−1 ⎠ 47  
* Dashed line indicates time-lag!
 The Original LSTM Cell!
xt ht-1! xt ht-1!
! !

Wi! Wo!
Input Gate! it! Output Gate! ot!

xt! W! Cell!
!
!
ct! ht!
ht-1!
!

⎛ xt ⎞ ⎛ ⎛ xt ⎞ ⎞
ct = ct−1 + it ⊗ tanhW ⎜ ⎟ ht = ot ⊗ tanh ct it = σ ⎜ Wi ⎜ ⎟ + bi ⎟ Similarly for ot!
⎝ ht−1 ⎠ ⎝ ⎝ ht−1 ⎠ ⎠
48  
 The Popular LSTM Cell!
xt ht-1! xt ht-1!
! !

Wi! Wo!
Input Gate! it! Output Gate! ot!

xt! W! Cell!
!
!
ct! ht!
ht-1!
!

ft! Forget Gate!

Wf!

⎛ xt ⎞ ⎛ ⎛ xt ⎞ ⎞
ct = ft ⊗ ct−1 + it ⊗ tanhW ⎜ ⎟ xt ht-1! ft = σ ⎜ W f ⎜ ⎟ + b f ⎟
⎝ ht−1 ⎠ ! ⎝ ⎝ ht−1 ⎠ ⎠
49  
 LSTM – Forward/Backward!

Go  To:  Illustrated LSTM Forward and Backward Pass!

50  
 Summary!
•  RNNs allow for processing of variable length inputs and
outputs by maintaining state information across time steps!
•  Various Input-Output scenarios are possible !
(Single/Multiple)!

•  Vanilla RNNs are improved upon by LSTMs which address

the vanishing gradient problem through the CEC!
•  Exploding gradients are handled by gradient clipping!

•  More complex architectures are listed in the course

materials for you to read, understand, and present!
!
51  
 Other Useful Resources / References!
•  http://cs231n.stanford.edu/slides/winter1516_lecture10.pdf !
•  http://www.cs.toronto.edu/~rgrosse/csc321/lec10.pdf !

•  R. Pascanu, T. Mikolov, and Y. Bengio,

On the difficulty of training recurrent neural networks, ICML 2013!
•  S. Hochreiter, and J. Schmidhuber, Long short-term memory, Neural computation,
1997 9(8), pp.1735-1780!
•  F.A. Gers, and J. Schmidhuber, Recurrent nets that time and count, IJCNN 2000!
•  K. Greff , R.K. Srivastava, J. Koutník, B.R. Steunebrink, and J. Schmidhuber,
LSTM: A search space odyssey, IEEE transactions on neural networks and learning
systems, 2016 !
•  K. Cho, B. Van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk,
and Y. Bengio,
Learning phrase representations using RNN encoder-decoder for statistical machine
translation, ACL 2014!
•  R. Jozefowicz, W. Zaremba, and I. Sutskever,
An empirical exploration of recurrent network architectures, JMLR 2015!

52