Deep lenig

Jntoouction to teed Rxuol nemal ntses:

odten called feedßusal-!
Deep feed Pos eada d ka, lo
netus , also o ten
netEA
Classise,cwech
"Plceptrons meany t s a
3enple binay
talkes meltple inguts, Porocoe Hem and Poodag a

Singte
Tue qoal
Otput.
a feed foTuoad neta k is to approzi male
Sone 92 o a
Enample: Po a clasyie Y= p*() maps an oput

Thesu uodels ae called feed fooald beCaute Cnborsabon

flous tougk ttu fanhon being
tnteme ddiatt copetaty eed tb olagia f.
though ttu
and Brally to tbu output y
eedforsaid neueal netwolbs aue callec netwoks beeote
.
typieally ryresented by capostig agetau
tlay ae
difs eent fenetns.
many cth a dlsected acyclie
Jhe eodel iis auate
funchong ale conpotid togetu
gBapk descaibing how tlu the functony f,po)and
Cramp : we uglt kau
(3) eonnected in a
Thee chain Stsuctwes
tle muat Commnly wseod
neural rnetks.
.Th ths case, f) is called lu fast lay
and
netwotk p6a caled tlu Bcond layes
llu clain
te ouuall ength name
mode! . Tt is fom this tuninobogy
t
ceep deatnng" olisd. is called tle
a feoclbrusalc netaole
The tnal elaye
output layu.
Dustng neutal netwlk training
()Tu trainng dlasa pouoet
ied by a
accompan'ecl
poits . tach eranple
traning
dabt f
ou not Shows u oleeire d output
dlata Calleol uddoa laye.
Taaintng
lach
a caleol
Netwdts a! neeal becase teey loosety
meerosuinet.
aspired by
cendentand feedfo n oasd neteka isa to beg
to ous to Oveieome
lireas molelt qnd congide
Ceuttt
tleis ellnitahoy .
logste egresin aad
 yaadient - Lacd leaaning
Tatooduch'on
muoh
a nua! netwk s not
and toaining
digpeunt from traintng any
olle macine
leatalg
eith grad'ent odeseent. butusen seu liheat neodlelt ,, nd
Ihe latgut oit ena nettk.
lineaily q a neulal
on-
9eelal netwky is t u non
beeone nan
unetn o
Causes los
Non-ineity
Neuial netwlk ale troined
funchn
tat du'us ttla cost
9radient -bayd optimizes than tle llal equahn
,eate
to a vey low vale model o tleu
folyes ued to train uol rearesta 2ecl lobal
Conv
optimization algorithms ut gusanto
Convagene (eg logiste rgresion o Sum).ony ribal
Convex optnaizahon convigei tartng om
Paranetey. Conven los
Stochahe graoien deseert appliod to non-
and
Punchons does not hae Jerch Convesgenco guatnte ,
)
Conve non Conre

. Co feed fotsad mewlal netwoke, trihalie oll wezit to

Small ranlon Valeees.
Sall poshtee
be eri'halied to 2
. The btasu may
Values,
 tle 9radi'eat de
he training phae is
descend tle cot feunch.
neas egrerion and
nlax moclelt train uaig
olely can train
Pel.
gradlent descent f e lasgee training nee conpltated
Compag
3tl be lone tbiciaily and iacty
tlea back-proprgas
Conputi Ue gradteat( ) ig
algouitem.
7o appy giaditat - baud leasnig
1. Tu cot furhon
eenits
2. The fom g lu output
3. The oph'neee
Cont unchon
Cot tnchin net be
leatning
. fo appby gratint - baud wlat output Bhold be obtaired
choen , ond additonally
alo ut be clonen
fuunchon es wsecd to trai)
. The to tal cot maxinmun
uocleen
meLlal mtutka ae tratnsd ung negate
tlu cost funchim is Sirply t?
Wellhood ie., tee toang olata
entoopy buteeen
9- Wkelihood (ooM
tleu alel olistot befon )
Qnd and
" The gradient
tbu cost feuncon muat be large,
abl 'to predit enogh fat) cendeinie lis
atui ate (become yey
Functons tliat
beece
tlu gToolient beos ey
ay mace tle
objectia
Small.
 tlu

hidoen nits & lu ot put ceits Satati

problm
Thenegaliue log- ieelhood hales to avord ls
k many modal.
i&tobeaAr eith aximum
s.1.) learing Conihonal

ikelihood
mqimum
" Mont oolsn newwal netwoka ae tarainsd wng
ikiood.
Jhis meany that tu cost funct'n u nogohue
neto is Boupy
tog-tikelikoood quvalently descr'bed as tu cDK - entrophy
oitn'betn.
training data and teu nodel
btuwen th traiig
Tu cost funchin s gin by
J(6) = - Eay wpate ogenadel (y/:) from
e cort functo clang
Speipie te Speuzie form y logAald.
nod to model , olpndling on
Boe
The pupanim g tle abou quadin byetally yilds da
" neoale poramells ad
tems tuat alo not depnd Dalel
Moy be oliseatde d.

to a Scaling factt g and a telm that oloes not

depend on O. The olscal cle dl cOnstat is badecl onle
valiance te Gaunan litibußon, oich in ttui's
(ae we chose nol o para mete
 tuiva lenca bteon
}'Manimum
arvfoutly ikelihood
we Paw that tlu fq
tsbnnan cith an output ditaibus
nd miainisat) Souased e o hold!
ea neoalel
The fqui'valnu holdy negardles
precict tu nean ttu Gaunsan.
Adv antage anehn fom
ma°mum lcliko
le coxt fenctom cost tanchn
" when desi wing
is that it ou! ttu betadennosdeldeigg
PCylaubnaally
each ioolel Spe byg
deteies a cost funhon loy P(yl).
lanu tsouyhout neeial neteek deugn
Dne re
cort furton nust be
that tu 92adled g teu
ia to Suve as a good getoa
lasge and pae dictabl enough
odqot'thm.
earning corditonal &tattes
1.) L PCylx;o)
nog a full Probability olitibton
Dutead one condihonal Patsbe
jut
we oBten ant to leosn
hae a poecicto (2;o) that e
Cranpl: we
predict t u mean
cuish to poeyul nutal netwok
we
Stßicanty able to epreut
Can tlink tlu euial netsck a
clar
beig.anchong ,
lth
funchon f forom featue fach as
letted only bey
clah dy hauing
bednes
Contni'ty anol boun
olectney
fpeef'e palametme torm
matheu than
 functonal gatu than jet a funhon
farcton bing a
A tuchonal i a fromuntons to real neumbes
rmappg to haue
Caamplu: we tlu cart feuneh'ral
Hut maps * le
s miimum ie an teu funcn

an ophrahon pooblam tth repect to a funchon

Soluing a matumatoal tool
Calleo Calceks % vaahng
va'atng is
duied etng calculs g
Ous at mselt pooblum
that Soutrg tu optaiyahon

yilds ofrom
infenituy many lamples
could train niiniaing te
alrg distibetton, funchan taat
tu tru olata-gen
funehon gus
mean Squased lrvo costfol each noyaleee
Parectiots ttu mean g y dlisfend Stattsb . A Second
Dibeent cost .fanchny gie tat
yarabine i
ceing calouly
quet oleríued
Valee
predists
predil4 tee media
tur
yeide a tancton tuat fueh a tuncha may
be alecibl
So long as
ttu family funchons men aboui e r a .
commmly called
Cnd mear absolute e i oten load
Mean So uaed erat gladient - bated ophrization.
to ts wien wud ith
 Outpet tunits

dny kin d 4 neial netu cnit that my be useo

hictolen nit.
Opeet udden featuey
Set
Thu feed fiatd netotk poridu te output
deyinad y hfl;o). The
Sone addibonal
ta
lanfomabn te
s tlun to povuia tak tat tlu ntotk nugt
conplute tue
feoteeres to

Payomm. (4auessan Output Oistbubng

basrel
hinea? Uhits fo Gaesian an oeutput
ceit
Seutput cenit s
is
o onlineahy.
tanformabien cuih
an ayin
t caled lineo Qutput eL'ts
produc
Theu ae dten lintal ueergld and
Giusn featuus h, a laye % bic em
. along wuth a
Summed postibl poodul lu mmean
chten eyd to
layes ae
Lin al Otput Ly
(autian alistoibeation: Aeoolelicl

6a conditonal Vaable bung

GautKian distybehn(pjadicted noalees
pas

tlu
Squaud ervot.

Covalace tle Gaseon too,

fenchan
covauana tlee Guaan be a
d to maka tlu
 7he coNianu mt he constosaned be a
posih
desinite matin t all inputs.

ufput layet, So typiaty

covavone.
to palametoe tu pare iUle
Satuad . tay
Becase lirlai eatt do aot algonitnyand
grad'ed bosed phinisation akgothay
Cpiculty M
digiatty B
eide Vaty f
optaiaan
be yed itl a
Oamo'd enita et Benoulli Qutut
jai'aba y. clarejeahon
binay
Zu Value g a io t s folm.
clases can be cat
PBoblern etle tto

2.
stibetn oue y conchnd n Single nube?.
jut a
distribebon iA desirsd by Ply-lf2,).68 tlis
" 4 Benale' neda bo pre cltet nly
Tle musal met Valld pno babihy
numbee o bea Valld
trntuwal [o,1]
. Satafying tlis corstraint equis Bome caeful desgn
yaleee
iea u t anol thoesholel

to obtain oralid posbabiity.

Ply=l) =max h0, minít, htb}}
. This dopis o velid Condional listo buhon, but it
gradient deset. hen
ueing gao'ent
poutd be haid to toain teng
htb moe otsie tle eenit itewal, tue gqadient
teu uolels otput th mspeet o (ts palaamety
becms o. Tis Canes iues S¢nce le haiaig
lgorithm toes eeolene
falamte.
stong 9 adient
A better appiocch en conits th

maimum ibeihood.

Tuis malcss cse Some

Sost ply feneton . UAeol cl'ret y fol

Broelll
Soptpls &s not tyeialy
tasky becaye it clo esn't g e t t
cistnbuled otput
ta output to tlu Co,1) range
funchin doe [o, + o) and
to a range
Soptpls Mops tu igpt mn-negahue output
but

is cued a
lken you want lo,1] orge.
it to tlu
necesarily ant to retnt
olont atbus.
thoulll" Outrt
Sotrnax units fR mul
pobabity olast ibuton
we epsyet aa n possblo Nalees nd
ibuete Vavabe ceith
Soytrrax functon. classigi,
ceged as tlu Output
Sohtmax funhon are abity listnkutan
to eprelent tee poobabliy
dlheent classes .
aide u nol elsct
Sottran funchons Can be wsed
betcwean me
ush tls nuolel to cloos
difpeent optoas ft Sone iteinal vaiable.
 Sotman funetn es le input
tlu St, Sotman hunchi
2]
o Coch clas
p2
Softrna l2;]=

Regulas'saon foi Deep lestaig.

poroblem in machin leasning a howw b mat
A Central
pasom ceee! not fuet on tue
algostm tuat eiu puborm
tronog data, but alio new inpats . Many stategis
eued in machin lealning to
Bptiectly deasad incMatd
ue tle dest errof , pottbly at u Pupn~e
Tduee
traning erel Tuee Stategies eate bron collachrly
mgulatisaon
Pasametes lom Peralties
leasnirg
Regulalisahion has boen cnd long bijou eep logse
liee lineal and aegrecon,
Patulatly in lineol modely
a paramte no
ft eiuits nodel capo chy by aodeng
u objechue Punehin J, desetng tn a
naly tuncton 1.
Vegulaind objchue

sd objechue (g los funt

"j(0; x,y): This i tle aigedoi
te loss and le Segulaizati=
Tt incol pogatas both
tem.
 Dalameles , tu iput laa
mean
tasget yalus Y" Tuis colod be
Sonetig
wsheh contot tle
ypapaamete, whith
ttu Tegulaisahin AlagelSmal oveil

inpact q tlu Penalhy tee oolel ratig

o Spalses wegles . wleieh coul
fencBorn
J) (9): The palam etee nom penalty
tu nuodel
poesent tue L l2 nolm
yuis term dis cocsages taige Valee in94

LPasannete Reguali ahn

close to tleo
woight oacay
ît is eommonly koon as

qutaisabin tem to te ohjeetre

re
ougin by adeing
functorm.N alo bnousn a
JS(o) 2

cnctusbod by analyaig
megutasabon can be functn
weigud ducay mgulaisdd objecRee
tlu gradiant % tlu
te
by TawtJ(w;Xs4)
‘(w; X>g) =
funchim (eg- toss fenetin
J(w;Xy) is te diginal ohfechve
fo a claijation &regmetion noodal)
 "equlazisahon tem
egulaisan stengtt kyppotomda
egulaisakan makes inpus
deta X a by pnaliq kaige
hauing
Valees .
kigl
Tui Caue tlu lgotithn o Shink tes

gs Parsetasly fos fatureg tla don't ha ue

gfoony prodishve poss.
L1 kequlalisatin:
Lasso leatt absolud
" L' regulasfsaton , ao known as
Shain kage and Pe lecton opesat )
mgulalipab'on is a techiq wud o prenet ovagttsng
tornachis leaning notel by addig a pnalhy qual
to tu abnouti Vales g Cte maguihude 3 co
eeonda
to tls lou funeton.
Thu to otse aa boss fanehon (usualy
neini e

un Bquad evro Cn gretin tat) wlen tu nocdl

-wlere
is too eonplu tt can ltaol to ougthrg
training olata bat poory
n c s n lat
Posom eel on

Rogutaviain addrk tha by adaing a penalty

boks funhon tuat dis coul ages age paametes valung
The Rugulais ed ohj eche tuneton tas
combines de
Ohjece tneton cnd Blu 21 penalty i gieen by
 o% funhan
(;*y) is tte aequlaaised los loss funchon ithout

gulaiabón cot u gvengt

Jppatamte tat
tat
poai tie bypepaiamte
el Vales < inpose
a

tag
absolate
punalng nte weegts.
I|Du, seprrsena tle L4 penaty
Sums t e

technique
Vale tlu glaiz ton
both and la th both
many ouieh Combaes tu streng
Colled Elas the Net,
metodsTle obie ctu funehon is iaan
pnalty (abiolste aleas)
di contools tu Sfregn ualal vale
ttu penaldy (Sq
cots tu
inu
Noam pernalid as cottorarsd optisaaton a
bo vieuud es
penalbu in ophrizahön Can
elaxabon netod
Nom
cuying dlu larangian
ptinizahon
constrairsd a tum o tue objeetie
Norm penalhy typzaly ovolue ading paramefesg o Soleationg,
penal'e lage Valeue
funehon that Shooteg constrairte.
qulaiam paenete
enforetng rgltisahon
t(Jte) - k) uklote wais
L(@,«; xig) =J(0; X) tthas ( s)
both
a
parobam mquies nodijgng
as Tacdiet
cgaiet
descent
dses Can be e Such
Vaious pooca du
Solulons wleu tle 9radient
Fndiag analybeal
S(e)<k.
 Pogihe a en cou
Shiok S (o) teesout lethoa it dop belooRk-
To
endetan lle controlns
tyeat tu poroblem os

Slaitas to megelised traäg

inposes a contraint n tle
constroint to an lo ball. i &is thu
ale
liuted to a reg an coAtTand
Ca noim, tlu uug ts ae

Kegulaisaon and unde costrakoed Ponblsm

to enge
RagulaisabiÝn is ten to machiae le araing
Poobles ae usell-legind.
megenio and PeAy equie
X'x, Oich ea ot
Tuis occurs o Vaane io Joe alereehon o
when teu ae fees gxanpes tan featus
Regula'sahin addyenes teis by iovehrg x taT, wih
gualanlees te matnk is invet ble
Data Augmen tafon eodl
mabe
a ache lealang
Tu best way ole data
on mle
bette i to tran it
te tent book fol moe iformahon
Contne refe
 udy nloise Robustne and Drectng Woise at la
teetboot.

weey bote labelad

.Tt a a macine leaaing apprpach tuat
maclint
ond urlabeled data traaing
fupeuted and cenfupeatiaedd learag
. t Palls betueen Sall anot
olel st lealns fono t s
he
daba. Jfructuss fsom tlee
patleng
taceg to Btract
" Then
enhance le leang paocaks.
nlabold data to
MATaak leaning

Parameta Tytng and paamta hasing

Spase sepregentat'on
and otee EMemble metods

Dropout

Prop, and anibold angent

Tangent Dtance, orage
classi' .
Content pleat efe text book.
f& all tlese