Part RAIN

A Intuition Encodings
ofect Roommates Apple pie g
Sunny o Food
Butiger

Rainy 9 ochitinen
91
Problem 01 Fined rule

weather Sunny Apple pie

weather _Rainy Burger

nisi point
l

g Tuning
I P weather Model Tippie
o
O O

03 ftp
fi l1
Rainy
foI

Burger
 Pplfood

IF
Tip
T
2 1
top IIP weather
Es xD

Prothom Round Robin Schedule

Applebee chicken
Burger
PIP
OP has to be fed back as

Network need to rember last food

oisengly
Permutation
 Above permutation matrix can be used as a

model to the RR scheduling which

realize
takes and nextday food
a
food as an
gives

n.iq

previous day food

00
I
01
has to be remembered

0107oz RTs attitude

0 1
3 1 3 1
 do not cook
Problem 03 If Sunny
and the last
just repeat
day left over food
If Rainy Coo autre nent day's food according
to the RR scheduling

with the previous day food for Rr

Now along
weather Depending
One should have to check the
choose yesterday's food or
upon the weather
food
today's food Along with yesterday's
also Mp
Today's weather is
an

Identity same dayfood

ment dayfood
Permutation
 Swim
same
food

Rainy
selects
mentdayfood

Ifttrisisttreiyexopgetedg
.co weaq.ru
Pabterfoodmatiinqpqpw

D L
t

ahem
TO
 i at

Merge

Imad
operation

AEdd.mg om
 I
1 o
O
0
I O
l
O O
O o o
O I 0
o 2 p
O I 0
O
Mase
1 I

F inert
old I
RNN
r
food
Food f weather
PART B Mathematical formulation of RNN useful for indening
Name recognition in the tent modules beopleinnewsart

Given any sentence find table out the names Tentabding

say XL Kz X Ktx
well
X Sachin played Shoaib very
egg 1stword 2ndword
firstenthmed

I O L O 0
Y
cop
KD Ya Yst YL Ty

ere Tx Ty TYMmaybrfgthmayofnofqsesamn.TOSeg

ist
X ilp training example

Txt and Ty 3 Corresponding lengths

1 Hot encoding
Wordrepresentation

dimensional 0 1
Iii binaryvector I Hotencoding
Our goal to get a label 9 for given X

Learning a mapping from X Y

 o

Xi f I 8
0
Hugesize as
everyBig Mw 0 74
8 Y
If
variable length
You i ftp 8
I O
I No parametric 80
i l STY
sharing cqsonkyyd.jo y
x'n ff
Ilp vector
concatenated

we require our network to read the Sentence word by

word from left to right seeing data as a Time Seg

1,47 qujBTharetheparameleist
7
1 949 7 Wya

a
tI I

c
O
RN N
unit Unrolled

xstxTHan
4xas.FI
3
 yjyfyf.gg
KHAO O ykssnpomm
t.IE
T T T T
see n3n4 x axe they are
u
also utilized
7
qso 0 miticized
act g Waadt 7 Waar't't ba
t 8Etanhlrau
y g Wyaa by
Gazsigmoid
softman
Li
Y
via 7
1 444 7
Wya
shared
2 weights
t

u
G
Rummy
Unrolled

Han
TXSD.FI xGx
 astkgfklaaaht ifwa.it't ba

hoodoo d xD xlo.TT 0,000 1

2Dmatr
Take a 100 dim

iiEnE.in nEsD
T

I
F

history
a
Matrix Augmentation
weight
Haa Wan Wa Matin

100 100 100 10 ooo I 00 X 10100

Egiontatanat
I

fast 10100 1 flesticle Concert

Wafa X't Goo xD

HI IF fat b
 Therefore our equations

att g Wa Ea't x tba

ight Gz Wyatt by
Parameters to be learned
Wa ba Wy by

tf f
KD KD Dem ME
sina.mil Tt Tt Tt

a x

Dj Dq
Fans
µy shared
 Wordleuel loss

L't fights yet y logy

C yctDdogfi yst
Senle
neeleuellos LCya.y
EAL't
ya
yay
L k 4K Li
yg
Tv
W
Tv
w
Tv w FEI

Tu Tu Tu Tu
24 He Hz My

Part C
Vanishing Gradient RNN backprop
L L the 1L 1L where toga
I
oftruechanPfIdietfd

I7gtabitt oIEHulfotiwii.E.io
i ceedmEnYaI
asffe.ioffiiI dHfhYidldaigtIatioI
w

See Computation Grabhfer

oostwButIsusinFtrmeaEghtcsd.BoonhstonanaGwen
off.GL nd at

dependent
1

ahhisorisdered
µebadwwd t
pathogens I network
and total

To Compute adofmphE'S
bfeq afteradding
If auffyspossible
 dhu

Ek Es Ei
rotor

it AdpdfYsdmuinae
qswy 8 t8 8 8 w

Am ME OI
d 52
on 8 8 8 few
me
Ost
d si

Ew I F Ew
offs offs off of I.EE
fsw fwI

fhwIffmmed's 3
fIutfhwtfwI
Boundedmy siffluis
i relatedtow
This algorithm backpropagation in times need to
which requires
Compute a drained differentiation

recursive and refetature multiplication Often

o s
oo
i
Since these
Si's are activated
of heme are bounded

therefore there derivates are also bounded

Sigmoid s Yu
tanh s 1

may enplodelVanish
 Wordleuelt
L't eights
ya y logy
C ya dog l ya

Sentence level loss

L ya
y 2 L't ya
yay
RNN faces is Vanishing gradients and
Mayor Problem
as the backpropagation is in time it fails to update

well with respect to some old time

This causes RNN

Vanilla RNN Block diagrams

to forget about
longterm eventsand
their effect
JEET't
act D Ffs ahtktanhfwaf.at xeti

If Ct
t ba
PART D Gated Recurrent Unit GRU
is the captain
Virat is a
good player He

w
Three persons
going with us and all of them
are are

www
n

Problem is that how to remember that subject is

o
11
Smgdur Pw
was were
11femaleMy
Male
He She

we consider our hidden state memory cells

to understand such dependencies and tract keep on

memorizing them till the point they are required or

remembered
Something more important need to be
Chthonto toggle
The 1 two Eton Theo Teo
is the Captain
Virat is a
good player He
I
E I CE l
mTN
E
g MALE
11
Remember he is
whatto
some bit value got set keep
 to it is
We need remember Subjects Gundy or

Singular or plural Lmatmate

Here It can be seen as the cell value
at time
Gammaumy It can see as the gate fattonsblou
u Some information
1 or

Fintan tmtkEgIq
0

Our Ilp's are at

New cue state

needstobecomput
and
AJ
Candidate update value
men

I tanh Wic x Jtbc

Neural network parameterized with
by Weibo
input D
as ut and yet Wuphattfate
 tuGbyak
Pu
ofkluf.it xht5ftbu
Neural network parameterized by Wu bu with

input as Clt I
and Xlt
This network tries to learn wheather we need to update
the current cell State or not

thungwmagthiminanummatthnighiitighi

SimfdifiedCRU
Cellulpdate
D
at The cut 1 pug est
un mu
retain
what needs to How much to
beupdate

M Fon1FTI
 Cht cult The all are
of Same dimentions
then cell update
Say doo xD
equation is elementwise mullifteiahs

Pu understands cell States bits behaviour twenty

few key are

effectively so
justtry to gate their
and tearing
as reduce the loss
neeaze.dz
pwooyrdeeu.antamanysebitmy
Full GRU o for gating

b
tanhGale to it x

Tu 0
wife bu

IT 0 W K't x b
Relevance another neural network parameterized
gate
D
with infant as Cat
by r br Itsy
It The E't d pay act
D

afterSeveral updates handle longer orange dependencies

vanishing gradient and better convergence Aft d
 PART E GRU to LSTM
GRU LSTM
netwamgate

Est tanhfw.IEodY3it5ftbe

E Itanh We a 7 7 tbc
4

on LSTM relevance gate

update gate GRU

signifies which
Tu 0 Wu
Edt zesty bu bits requis
Elt and est i updation remaining bits are not
are
weighted by Pw Changed Modified

Update Galt LSTM

4
Tu o Wu East 7 7 bn
mfies whichbits need to be update
mny
Relevance gate Gru algentin.IS

0CWrEcstYzdtY b Tf
 Forget Galt LSTM absentinLSTM GRO
A

17 0 Wf fast se't tbs

LSTMt will learn a forget gate separately Basically how
to
we need
forget from Clt 7 previousState

Output gate CLSTM absent in GRU

D
zesty b CaaIt
To F Wofact
at gated by Po is the final ol P

D
cht2Tu EltZg_puJ.o Et

updationes TM
D
Ct The Est pg Et
final o P GRU

ah D It

att l get

i i

ii

at Toa

It
tf
PART F Review
C 21h30 123070