Scribd
Upload
26 views16 pages

RNN LSTM

The document presents a weekly report on Recurrent Neural Networks (RNNs), highlighting their mechanism of retaining information from previous inputs and how they differ from Feedforward Neural Networks (FNNs). It discusses the advantages and disadvantages of RNNs, introduces Long Short-Term Memory (LSTM) as a key component, and outlines the backpropagation process through time. Additionally, it covers various types of RNNs and their applications.

Uploaded by

Hoàng Tùng Lê
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
26 views16 pages

RNN LSTM

The document presents a weekly report on Recurrent Neural Networks (RNNs), highlighting their mechanism of retaining information from previous inputs and how they differ from Feedforward Neural Networks (FNNs). It discusses the advantages and disadvantages of RNNs, introduces Long Short-Term Memory (LSTM) as a key component, and outlines the backpropagation process through time. Additionally, it covers various types of RNNs and their applications.

Uploaded by

Hoàng Tùng Lê
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 16

WEEKLY REPORT

Dang Vu Do - DSAI

2
What I cover in this slide

• Recurrent Neural Network (RNN)


• How RNN Differs from Feedforward Neural Networks
• Advantages and Disadvantages of RNN
• Long short term Memory
• Transformer
• Applications of 2 aforementioned algorithms
RECURRENT NEURAL NETWORK (RNN)

Recurrent Neural Networks introduce a mechanism where the


output from one step is fed back as input to the next, allowing
them to retain information from previous inputs.

using the same


parameters across
all steps, RNNs
perform consistently
across inputs,
reducing parameter
complexity compared
to traditional neural
networks
RECURRENT NEURAL NETWORK (RNN)

Feedforward Neural
Networks(FNNs)
process data in one
direction, from input
to output, without
retaining information
from previous inputs.

 suitable for tasks with


independent inputs, like
image classification
but struggle with
sequential data since
they lack memory.
RECURRENT NEURAL NETWORK (RNN)
RECURRENT NEURAL NETWORK (RNN)

Recurrent units can


Recurrent “remember” information
neurons from prior steps by
feeding back their hidden
state, allowing them to
capture dependencies
across time.
Key
Component
s of RNNs

RNN unfolding,
or “unrolling,” is
RNN the process of
unfolding expanding the
recurrent
structure over
time steps.
RECURRENT NEURAL NETWORK (RNN)
Types of Neural Network

One-to-One RNN One-to-many RNN Many-to-one RNN Many-to-many RNN


Back Propagation through time – RNN

• S1, S2, S3 are the hidden states or memory units at


time t1, t2, t3 and Ws is the weight matrix associated
with it.

• X1, X2, X3 are the inputs at time t1, t2, t3 respectively,


and Wx is the weight matrix associated with it.

• Y1, Y2, Y3 are the outputs at time t1, t2, t3 respectively


and Wy is the weight matrix associated with it
Back Propagation through time – RNN

The error function:

• Adjusting Wy At t = 3 , we have

• Adjusting
Ws

• Adjusting Wx
Long Short_Term Memory (LSTM)

• [h_t-1, x_t]
denotes the
concatenation of
the current input
and the previous
hidden state.
• Σ, tanh are
activation
functions.
Long Short_Term Memory (LSTM)

Forget Gate:
for a particular cell state the output is 0, the piece of
information is forgotten and for output 1, the information is
retained for future use
Long Short_Term Memory (LSTM)

• The addition of useful information to the cell state is done by


the input gate
Long Short_Term Memory (LSTM)

• The task of extracting useful information from the current


cell state to be p.resented as output is done by the output
gate
Long Short_Term Memory (LSTM)

Overall
THANK YOU
!

16

You might also like