Recurent Neuwral NetuorkRNN

RNN wrks on the pinctple of Sovnghaoutput

a portiCular laueY and feedtng this bac to the

tnput in OYdev to predt ct the output of the layer

St-) t) (t1)
hlt-1)sir)

hlt ht+1
C

Gud

3
( (t)
Y ulov t x(s +1

Fully Connechod Recunevnt Nettval letoork

A,BC are the Pavamete vsof The, Netoor.

Here is the input lhyr, h 1s the hicdden

layev,
layer, and
anmd
s the output layer A,3C ore the

netubrk, paramelers uSeed to, psove fhe butpud of

S 29
the mode-t any given. tíme t, the Cument input
s a Combinatlon of input at x(t) and xt-1).
The output, ot any qiem time Is fetched back t
the netooYk tb impYove 0m he outpus 291
ht) neuw stote
htt: f (n(t-), 2(t) T unctiom th
oko2 2lo 2tdp bvr porPoramettr C
why RNNS
why RNNs? oD so ht-)0ldstaB
sa x(t)
(t) tlp
lp vector at
RnUn were Creattd beCouse therd
time stemt
were a few Sues in the
97
feed-foruord neural nutoork ANNS).
2o0dhp
Cannot handle Sequen tlal data
Comside rs only the urent input P
Connot memorize
ovulo
prevrus nputs
The Solution to these ssues s RNN. whch Con
andle Sequenttal data accep tt the uwent /p
,

dato ond prevously

recelved pt and alßo, Can.
memolze prelbus p's de to thet intmal memory.
otuo po
mife22l) 9po
 2.Explodlng Gradlent Problemt
For pmetimes, we omly need to lbok at reCen

tasSK
intomatm to ptrforn the present
NLP EX- lhen we t to predict fhe last word

The clouds
The are in The.
KNN Coan able to prehct the wd as SSino

he
The gap behioeev) the re lavant m frvma hio md

the plac that its needed3

2
s Small, RNN Co
leavn ho use the post infoma tlon.

ho
o9oan2(Yo Yy 9h2

A nh
9n9p bin

15ut There are also cases where ue need aUre

Contyt diol2nort 9t0o-

NLP EX Lets tring to presict tha ast wod n

The txt
igreo up in frant. SpeaK thent
rench
The ReCat infomatim the nent od s proa bly The
namd of
lanquage butf e uant which
a

languge, fhen we nee d the Conttxt of france

rys fi

2 fro prevous Sent na. Heve gap between re

lavent
infpratiom and the2 place it s neeckd is Very large
 Unfovtunate ly, as hat gap gros, RNWs become

unable
unable to learn to Connect' theonfomatlisn.

h
V Dopfeysf,
ho h2 hr han

HA
Very lage
ap
Heve Comes The legend. LSTM RNN

LSTM-RNN
Netks
Long Short lerm lemory Recurent Neural
uSually Colled "LSTMS" are a Spe Cial kihd of RNN
Capoble o learning long- term dependenCtes.
ISTMS haye ability Of reme)bering intormalion fo

long periocls ot timne

All RNNs have the fmm of a chan of repeating
modu les of heural nethwork n Standard RNNs
YYp
the vepeating modu k will be a
Single tanh layer.
shor snid (htt1)

oAoh Hanh A
OT do
lop? ) RNN
LSTHs alSo have this Chalm Itke Structure, but
the epeatng modle has a ldferent structu
There ane four , interachng in a veny spe Cal lau.

hu h

me noty Ce

A A

Jo sloiogo)
NOTatibS! hoosgs h-pol pxinvnsl
Tdiomototf
O
Neural Neturk pointwise Vecto COna ttnate
layey operatim 9o u
Trong fty
Copy
T bioopd2 rorulso logsif 23bom
n h abge daqyoam, Rach ne Codes an ente
Vec ror, hron the Dlp of bne node to the of ofhevs
inputs
Sttp-by-Stp LsTM lalk Throug
The first Step in ouYLSTN 1s to decide tuhat
ntor matt bn weve qcing to throw away from he Cell

State. Thisdecistm s made by a Sgmold layeY

Palled
Called the fovget gatt lauer, t loors at he-1
and tand Outpuy a numbey betueen 0 and I
sqmoid) for each humbey in the Cell state
C
ACompettly keeptks 0Complekly 9et rd of
this
prev-
C EX-Lets Condt er lLP exomple,
t t may inclue gender of the
ht- Present Subjet. Whewe See a
hew Sublect ue Oont to
XE forget
Hoqet-gate gendev (old Subtect)

The. nent Step 1s to decide whot new mformattom

e've qolnq to Stove in the cell Stat. This has

tuo par. Sqnold lauer Calledn put qate layer

A

dectdes which ValuuS we'll upda e. Nent, a tanh laue

Creates a vectoY of new Condhda te Va lues, C that
Could adde d tO
be the Stat. In nent
step, we
Combine the se hwo to CYea te an upda te to the State.
,t ef bet (1EN t we} ant to add The
d gen dey 04 the neo Subtect fo
Ce he Cell Sta te, to replace the old
6ne e' vYe forqetttng
hA
(he.,]+b)
tan LA, he )be)
 Nouo we olll upola te the Old Cell State, Ca- into
tnto the
new cell Stat C nle multply the old sta te
t,
Yorqetting the things we detlde to foget eavter
then we add ixC, This is he new Cardidafe
Va luu, Sca led by how muth we dedide d fo updoy
each Stoatt Va lu

2 J929

ton
CeCti,
L
Ewhevre we'd achually drbp the infoold
Subect (gench
and Odd The hewntomotlon as w
decl ded prev
Fiolty, we nee d to decide ohat were golng to Output
Ths blooip.2
dutput wll be based bn ourcell staie, but, ll, ke
9 ohgw9i 2dlov
a lkred Ver s pn. Frst we Tun. a
Sigmold layer which
dectdes what ports of
the Cell ta te e re Qoing to
f2 Jrg
Output. Then, e put the cell stat tmmgh tanh
to puth Valuus tphwen
-121) and multiply tt by P
the Sigmoid
gatt So we Only lp the
ports we
blo vit s) dectded ta

pyTo0 900

h
h he Oxtanh(Ct)
EXl N L P
Tonn
Tohn phyed tremendous ly twell ahd
toom fur his tean. for his Corrtributlon S, brave
was aaYde d player of The math.
There Could be
many Chol Ces fo ernpty spao.
The Curent ilp brave is adjective, and adjecfive
descYl beS a
noun( Tohn). So, "Tohn" Could be the
be St Outrt af ter braY.