lOMoARcPSD|48123243

Deep Learning Handwritten Notes

Deep Learning (Dr. A.P.J. Abdul Kalam Technical University)

Scan to open on Studocu

Studocu is not sponsored or endorsed by any college or university

Downloaded by Ashwani Singh ([email protected])
 lOMoARcPSD|48123243

Dutinction bten ML 4 DL:

* M
wth dotawts
canpiaiady
alentnde by mathemahel anyis
Aclig Csomalizion,tanddia
Faatuie Fealue lonnig
ML kane to manualy cary Featue Erg Bul
DL, Nenal Ndkot.
Principle Camponent Anoby iu : Dimengienal padnction
teabua 2

Tabuis Qttiono: Con we do iressne procae agint

Yes, Can cohnet the

Trtiuo
Featue Etaction: Crealing eatuea trm the
The ectactd eatue e Aut endu, LoA):
then Aooded ML gerthe.
Slaction:

hoilsize er)din ansign

a lacsiue teakuue iinakion Cere),
Lase /Cidge Paqulaiisabion
Car

talle daciiong them AT

caled AT. Thiy Qoei nat rsire human fajation
"Neel Nelurts e

AI : ML: DL

miitaleu

Tendortta+ Fronaer
orch
Ntort ’ Tha
Nodu
Corrsponde to numbe
tAe dalaet
Dutput
>A NN
chaice the indiel
car tale y numle.

's cheica
Downloaded by Ashwani Singh ([email protected])
 lOMoARcPSD|48123243

n the u then
more than then
a
prolden
g
the the op

core ction. (reloren e or a portieulat teau).

ropgaion S- Sonmaion
Comhined
torthec).
S- Ackikonunction
DPL

Val

are
tandomy t he
tc: thiu kauation u bed to creat a line for 2-D data.
data
linea. But
Non- near n nat. No, to
NN modal Acivakion
tundions. an-linaiy
: Tha igomoid reslt
tunctinThis to prdue

Ackivation tunction 4):

modal.
: Noco he
’Baukud pro pgaon match
trat set af prodiced
uelws, we do BP and tusk t dharg 4 bias vaes
rul. Thiu
wightfheerr rat
ko blaun on

Nesal NetwotG ale alio call,A Uniaed tnction Arpnzim

* dha relei Aconept aleul. The
boded Vahios factors Auch
vahiou ofhe factors acl as
the nlotinhip bataeen
called chain rule.
* Gudint
ueghtAmall
ua

Cast byncbi
-Downloaded on Singhprmd
Ashwani
actu): Thie u the eror
([email protected])
 lOMoARcPSD|48123243

Binay
* More hen
clauitiection.
2 Loses, Mulk-doslitiekon.
+ Muh a: uttible kolla or MLce, we we
4) An 4 cet. co e more
han 2 lole
Adintion Fundens;
Tanh tunction : Hypea lolic caie buncbion
2
-22

The anh uore mot

opimizaton aasie
uith -ve unles
More

’elU tunion: Most Gunoialy irplemaled in

the thdan
Does nioe toka
unit
Amall
asback

NDmal pelw. Dang-o, ).

tor x20
when haue mullbiple laue. I u
whicl
(o,).

whele
o, tain e

Colection a! Can importbd

an

want
braug c) Ten
for :lighpilomaná ramuical conputalion, ,Tar
erod : A tigh taue that
ApI onfop ol enyoro), CNT te
L Tenses: A mulki- dimarsiona wheee dka u Atore

Downloaded by Ashwani Singh ([email protected])

 lOMoARcPSD|48123243

Hoo many bimes the etire

Corree
baie. thiu ime To holaL thiu
time coMUeng bateh the datact
cleto n
ieant
all the colnne n the
max cue', None)
((- used fo eaine whathe a
ofe Male calagoiad

Csnseat female nto o entire dataat

We Can inszad ot alal en
that naode tsonafomed
Avc- AoC: Area hde Gue- Racmive Dpanbing charecaatie
ony, orploted
clasakcalion prolame.
tr Falae Pocitie late C True positine Co)

Poie
Troe
Some ohae
cQarifetin Agp. Csum)
hat
Rardom
Jhe poc Cue duturiny
the this proben ctalenent
more nt
Falle Positive panle kion Ccoiwsisg'
townid
metti. area

’ Douk: Similau tor pandatttos all he matkod lit in Pandal

Orgonabne dask outol ponda a it on hanAe
data
Convo lutional Naucal Netorte: Ma
more auanel than ANN' yed "when he daka hal a

Apakial auctus
O- 256 ale
PGB 3 chanel

vidaos dalosek.
when NN,
Aantifieu mutie
Contou Conto t
Core prin ciple calld Corolutron Operstien. IE u a
pabens an the
nslurort Can Male or

Amatl indo to an to estact

ing
Downloaded by Ashwani Singh ([email protected])
 lOMoARcPSD|48123243

Hat acrosS

t a me a action of
called kanel.
The i ze, data Re
laun Outyut
arid Gavid ducidd by Ihe
C3 x3) CNN

(nxm) axn) (m-n+) Cm-nt)

enaroludon o, the dinenion
problam For a
onage duca

re eutput. CNN AA

the
Featue Ertalion
Featue acoss he

have Caruratuona
O

cperaion à
entre dekoet
The open Con bo hoe Many ine
aihirg the ho apochs
Jaains ntire lataat
modal
n
medlk pertotmane o- bime

n
ropresontd mal times,
lotne

ot the beoe alMsnt. Thu caws

ontation. To aid
nenen
colun rows at

1
make Au ha al th
Aamentas

2 rDws d

Irgortent raalons

on t
’Feokuu epreentolin pordeu.
Downloaded by Ashwani Singh ([email protected])
 bchia ed: fo redua
lOMoARcPSD|48123243

he spatal inension Cundt

}aabude map uhie rekainirg the mpotontt

more obut
Mac a: Talce
matig
he marn a

Taller the Lale

a and thu
matu
can Crata constionel oeralion CeN) ond
add ANN Ao

Conieala a 2D This à becoe Arn's

Comlie ANNy we attan
NN COnhecion.

3
2

tant
Neal Ntwore. Alrgsid Nawl
Netotca relall
dinenicnaldata,
meltod inside ansthet mathod

* Dabe aigmentatin Techniae uud fo alficially

aaig datost by
data Ths

haue
the

Stdes in
bre: Np ponda, Ten ols Uaa), lanas Aataat : MNIST
’ Split aain, eut: Dirde nfo, 4 caugone:
mait. lacddata )
’ Haden te data : Conseit malk-iman ional'npcn
’ Normmaie the data: Normali He data bo tat the data tointe

neicl Jata.
Downloaded by Ashwani Singh ([email protected])
 lOMoARcPSD|48123243

ho heultn.
’ In hden
No Nauupn Nawon
oatput epoation,
* Actkiadbn tuncliond.
Aivation Funekon.
atumine ho much npotane
thal
teened ntust
the outpt. weght halp
Jaon trom data t mke more

mest liont ntamation. it to

* Bos: Tt helpe the netot make belae predicers by

Considael. to te
non
* Aebation Fention: AF popose
networt, whiel neans the net rprasant
the data.
patens

a vale betusen
-: Output
signoid:
prediction, CFo Brey Caleulation).
Tand: Outpet
Centeu oand .

à colle Feand
* This entire pcus
eialle
hauael netwok
callad as Lass
fucti. CAcbe Prodicd)
dose to 0.

ase irordo relue the Lest fuctin.

+Optimizs:
to miumize tre
’ sto chatii Guradiant Delant CsGo)
’ Adam CAplhy bie Momant Estimatin)

Gradirt
Downloaded by Ashwani Singh ([email protected])
 lOMoARcPSD|48123243

Gradient Dodcent: This y a fundarantel

minimige the
Naual Netuoe.
Loss functien

Slope Sepe for Ce) Aope, wnas ual.

’he main aim à fo
to tha

chain pule Diterentiain: maun

thedL the he

loss wr t
chain re

ol deatiue
Similaay
Ju dodw, lois

Rone
g Grient Prodem: Common prbn in DNN,
those 1t shen
Aay un dion bcome Amall bactunla
notot thu causs sloee or) Atalad
VGP
* ReLy Aelivabion 4 Bateh Normaliatien

’the taks plae in Siqmoid Aetintia, tuncbia

Tke doiti'e Eack propgtion
Ackivation Functisn:
Sigenoid Fun ction :
(te
CNct 2ao contuie)

’Darvatine:o to o2r.

¢ tanh Function: (2eo-nie)

Meaniq: the cue

*Relw Fun dion: max lo, x)

’ The pelormonte u the best in Pa.
’Deu vate o o |. ’Net zei0 cantie.
the Ilp y -u, the ReW uil die.
Downloaded by Ashwani Singh ([email protected])
 lOMoARcPSD|48123243

Reli {2) mare (oolx,).

Aolwes
Eponenia lineot Cnite
oheasse

’For Einatg daifdan;

’For Mak dan dauihcaiors

Bgien prolslms tia Actiion.

Locs funchionei The oss fncieni
’Mean Eror MIE)
Absote Eror CMAE)
’Mean
Loss
* Cost Vs Less funtion : tha losr

unctin pidu

The taing proe minize the cet

al
sigle studont Los.
ost
Cross

coss
optimizars: Rala preious poge 4 Grradient Dacert or more dokail
* Epod: Dne faosd 4 =|

Grrolient Jocan
Ous te entiree
frown teaiing datasetat eac

more
one

aoinig Epodh
point FP
Thrtion |1
prcaig
Downloaded by Ashwani Singh ([email protected])
 lOMoARcPSD|48123243

Oleome Normal (
* Minii Bateh lo prodm
mini atc SonD.
Batch siza (ooo We (ooo Dkapoinda
FP
Daba po inb Ibuation |
BP

wil a belle.
’Reaoute Inkanse, Loy inme ComipleriH Conueg ne mnia
’Wodt ole
5untlaue the SGDas the mort
Mi Bat c sD with mometn:

noue, we Nbue Noue

momentum minimal Noue
It Amoofhen fae GD: No noue

Gtoeal Minina.
heobal Minina
Uled to nore
Argconuidaing pit data
reant data pointa ahile Ati t
remoe noise

danivatve patmelea

Delcsnt: Tem, Qasouce

laaat
GD:Smoothen e
Mui Bote itte moment un
* Adap bie hradient Desant
for porsmota
whaie

This tale get

dacreeing
data podE teache
he aobal minima

Monartun t
the rate tor
ond male ye of honahtn to acealuat
Thy
Fonal

Fomela
Downloaded by Ashwani Singh ([email protected])
 lOMoARcPSD|48123243

ANN Tndanantakion :
Panda, makyloibsitit-Aeatn

indapndert eakus.

into naunerical data Featue

* The next
es

Akutene
tzntovtlas. katni. ayau onpot Jenie Ew. Aition
- Laatyu, fRe, Fun ctio tor
Drpout. Ous ANN
fo
4
ope
aanca. a conurinent
The * Dane: A Donye conreethd Jayea n
Dairs ün

prourious asye Poone ot Dainse's

’ Unitt : No at ene in the Vaye
he IIpAota Firt
NN to pesent

a &et naueoru toaion

’Impous Gnalien

data
* Initialize An

Joaitad (enie (unit =M, actidion 'rela)

enata a

Metiu =
Cos itie compile Copimizs ='adam',loss :

Downloaded by Ashwani Singh ([email protected])

 lOMoARcPSD|48123243

malel his toy clanif ftC bain,gaàn, adaor splt. oss both te,
: Lal Aot eoche tooo. At cre cpochs teoo)
achac certa
males the moA re cution sops atE
e
3han
Ateppad inpig retlt

Oleck Bor Model whitk Boz Mad:6

le nt aprekel
baon ()ondtondg y
Daep Naal Neleuu: ANN, CNN, RNN 2F etc
procsk rutaonaparert, ndautondall by humanl
aay Deiien Troe4, Cincal
Consluinel Nestl Nett
In

’RaB: 3 harnelu,

48
O-25

* Min- Mar A Normalization echrique sed to aca the piza

to a
lattea wen n Similal kealeThis
ansu that do not
(-4) o- 21 convebd o-t
Some iabe On cach 4
Fit/tanl
-2
6xbo Funtior
m
2

Conuoluor 3*3
Cm-n)
Oparakon
(6-3)*
OxO+o|40K-2

OxIoxI4OKD

ued Downloaded by Ashwani Singh ([email protected])

 lOMoARcPSD|48123243

* Mae Exbaa more

uridoo
windo
indos the Stide on

6b Min, Maq
2 &
4
1

3 More cip Tg
Tmplamentation:
modd Seqantil)

modd = madale. suqyonkl)

- (s2, B,))
Modl -od Clayme Corw2> (32, 3, 3), actation 'rlu', input. shepe
Neaf

ime- Saia data, tet or hae a ymamorg

to retain intomton trorm
shee onteet ovde mattu. Traditionl NN Aon't remehe
podt inpt.
NLP.
Uhe ofp u
Nauncl

Santinent Anays, Auto gt

Kequanca Modol
important. For this proen, notmal nenel netoa conot e
ANN:
hen We

* Too mueh cenptotin

(2a) Statnánt : Dhoni ú a

Thani
Neuons
Timetong
Not: The Aoene
Hidden
Dho
Veckorised

Contains Dhoni, u
Downloaded by Ashwani Singh ([email protected]) Cntaie Dhai u, a
 lOMoARcPSD|48123243

Encade : Canyet
#
pri,

edate hidlan atati: Ston nto

ureat rokd
. Preict the o(e : Gnenate ped dios (y nert oordi) baed on
hidden data
lo: Compae pwdutione b abud atpue
Tterat: Continue tha molal ra
how wnrol a
* No matt the
that we hae to ban.
more

Grodient proldem.
radicnt proben:
unth EGP
Tnput, AF

Wo
AF
t

the p.
AF
The cyl
n er Esçledsg Gdint proldam.

for th Low

Loss
Fundion

PU.
Paranet
qmdint containu a

to praant
Caradint probsm:
urith
J to os. e haue itzadion
ci.e) t-3
AF a val
t
AF

Downloaded by Ashwani Singh ([email protected]) Garadint Problam.

 lOMoARcPSD|48123243

omel
Can

inplode
maun ida bchind hos LSTM

Aarme conneion tor

2
aant
mate phediction alout fomoos
for shot-tum mamorin. This muh more

unit onlille Ubnilla

Uneice (he offee UTM Atiiation
Acbinlin Func ion more
Fun ctione, Pale Ae urou(REF- 2)
how thee

ramemel.
Ahould pdt
’ Out ut hete: Delremines how e Ahould
LSTM

the Tnitil
nautdl steat.
CHddn-Stot]
The bciuey cle used tor KTM
Mamones
at eacd
or sach
The updabd short -tam mamag
LSTM
Prprtlentabian
Surm

To understandhow the Long and

Short-Term Memories interact and
result in predictions, let's run some
numbers through this unit.

270 2.00 K141 Short

Long-Term
Mermory

Tritidy Now that we have plugged in some

nunbers, let's do the math to see
bo 2, CTM to l. what happens in the first stage of
the Long Short-TermMemory unit.
Surm Sum
2.70 2.00 141

Short x1.63 166 094 1419

Term
Mermory Input

Downloaded by Ashwani Singh ([email protected])

 lOMoARcPSD|48123243

Long-Term
Memory

Now we plug the x-axis coordinate

into the equation for the Sigmoid
Activation Function...

+1.62 (1x270)+(1x1.63) 1.62 =5.95

Sum
2.70

Short 1.63
Term e5.95
Memory Input f(5.95) = e5.95 1
1
= 0.997
Long-Term
Memory
2x 1.99

0.997
So this first stage of the Long
Short-TermMemory (LSTM)
unit reduced the Long-Term
Memory by a little bit.
+1.62

Sum

x2.70

Short 1.63
Term
Memory Input
1

Long-Term
Meremory

2x1.99 Sum

% Long-Term %Potential Memory

To Remember 0.997 PotentialLong So let's plug the numbers
To Remernber Term Memory
in and do the math to see
how a Potential Memory
is created and howmuch
A of it is added to the Long
0.62 +-0.32 Term Memory.
Sum Surm
2.00 K1.41

Short 1.65 094

Term
Memory Input

Long-Term
Mermory

299 Sum

%Long-Term 0.997 Potential Long

To Remember Term Mernory

. .and we get the y-axis

0.974 cOordinate, 0.97.
40.32

Sum (1 x 1.41) + (1 x 0.94)

141 +-0.32 = 2.03
Short 0.94 e-ek
Term tanh) = ex +ex
Memory Input
tanh(2.03)= ed- e203 = 0.97
1 e200 e-203

Downloaded by Ashwani Singh ([email protected])

 lOMoARcPSD|48123243

Long-Term
Memory
Bum

Potential Mernory
Long-Term o.997
To Rernember
To Aemember
.and we get the y-axis
coordinate, 1.0.

to.62 Sum
(1 x2.00) + (1 x 1.65)
+0.62 = 4.27
K2.00

1.65 Gote
Short
Tern e427
Memory Input
-1
n427)
1

New Long-Term
Long-Term MemoryA
Memory
Sum
2.96
% Potential Memory Potential Long
% Long-Term To Remenber o Term Memory
ToRermember0.997 ..and we get a new
Long-TermMemory,
2.96.

Sum
5 -0.32

SUm
x1.41
200

x0.94
1.65
Short
Term
Memory Input

1
New Long-Tern
7Long-Term Mermory
Memory
2xH1.99 Sum
2.96
%Potetial Memory Potential Long
%Long-Terrn
To Rerenher 0.997 To Remember 0 Term Memory
To Remen

5um Sum

Short
Term
Input
We start with the New Long-Term
emory
Memory and use it as input to the
1 Tanh Activation Function.
New Long-Term
7Long-Term Memory
Memory
2-199 Sum 2.96
% Potential Menory
Potential Short
Term Mermory
%Potential Memory PotentialLong
% Long-Term o.997 To Remernber Term Memory To Remember
To Remember (0.97

+.59
o.990.99 Outpt
Bum Sum
4.38 0.98
-0.19 New Short
Short Term
Term Memory
Memory Input This New Short-Tern Memory,
1 0.98, is also the Output from
this entire LSTM Unit.

Downloaded by Ashwani Singh ([email protected])

 lOMoARcPSD|48123243

Tranafonarai CuegKor m, Tiranfeme modla

that
al oltantion machanim" them
Hfeont pote them

Aata, wnthout Aalationhipe

kuditional Lerm
hundamantally boed Tanfomt. hot So,
nuel Mott

* lathar kon ndom nnla, to (oord, Can

Aimple u. Neton to
that t can the Contet n Aataat
b ogtimie ratullig in
bhat Con
sinilad wor
* Thiu makat the
Enbedigi Danie, contlna
op bue t semanti Aimila LSord doa
othe n thia

"Quean' hes
* word 2Vac: A achniue that ee a naeol netsort to
loords boel thin contert 2

main Metho:
on

contort wordd
Dhoni a

’
Glam : Grivan a csord pedi Contit

(Dhoni)

pradlict e otes usord.

ye coe we je (ocos a Acivntion Functions to mop a
Cutain wod

RNN
Genrel
ENodou
Tersforms
Hen, fhe words
I poued

Downloaded by Ashwani Singh ([email protected])