Overftking is hen o classificn 75 hu Anatning lade don Highly Such o classifier wanks well on-thy ‘Tnaining dasa tut nat on dependent -lesd dada. 14 is 9 genera Porsblem -that Plagues als Machine Aeaoming Methods. 1 Because 6 ove iting , tow ermn on ‘Inatning dade andl high eamen on deg dale, 7? Ovorditing occurs whi a madel boegins to mumani2e Aaraining dade aathen Shan ecsining 4v Gemendize from doend, r> ihe mone ddticud a ei Hoven 1S 4 predie tu mosu noix SXSAS IN Past intinmakan tha ned 49 be ignoned. Thu problen is Aelermining which pont to ignone. b> averting generally occuds whut a mode) 1s exctesivds complex, | such as havin ‘tov many parameters selatve tothe number de chgervakons. PP We can deormint whether 9 Predichve model is Undenfating On ovwrfidting thy ‘Jodining daly by Josking o& tu Poedichan €AN0) ont ‘Daining dala and tu walualion dada, Balnad Reasons fon oventul ing I Noisy dade 2. Training sek 1s ‘tov Smal) 3. Longe number oh feature.Methods ts avoid _overbitirgs A Reduad Enver founing = ~7 Pruning a deeisan node consists db Semaving the Subdnee rocky at -Ihak node, maxing Ha leak nade and assigning Ht thu mast commen classficakon ci liu training examples abfiakd cuith that Noolt “7 NodeS Aru Semaved anly Ye the Desulfing pouned |e perdooms "6 danse “than “the asigina) oven the validahon Seb- -? This has -the eBbet that any eak neds added otc coincidina) Sresulannkes jn the braming set 1s Mealy to be pruned becouse these Samu coincidiades ary unditaly ts acca inthe Valichion seh. Rule post Powning . Ware FAT feunin 7 In thu pastpouning , id removes subtres fram a dats grown” tet A Subinec ada given nede is pruned by Sumeving iS bavanches and deplacing it with & ea} . ths ead is Labfed witha mast dnesuent elas, among the Subsnee being ouplaad. 7 Rae pas Rowming invanes Alu daltwing sims: 1. Sinden the deciston dour from Hy Anaining Se, gewwing thi Tee Unk) thx donning chk 49 as ett aF pyssake and allawing Ovenfilting to accu. 2. Gnver the tanned suc ints an esuivalend Setat ovals by cneahng one sult -danr each path -frond-tty Sest rocked Lead Nady| 3. Pour (9eoal%) each sult by Semaving any poedschons Phat Sieult 3 impaoving HS eshmatd Accunaey. 4, Som +r pruned guks by thin eskwated dccanacy, ancl consid) Hum ‘inthis Sequence syhen classy INQ Subseguer (nstanc .- 0 2 Sreonponaing rhs valued. Aone wv Our initial débinition 6b 103 is mesinickd 40 aMoibules that Foxe ona discoele set ob values. 2 Fist jthe ‘Tongt abloviiule chose value is poedicled by Iy Jeained Anee must be discoele valued. 7 Second, WM OMonbuks Jest Inti decision Nodes dk The Aoec Must also be diseaele valued This Secord nesinicton can easily be nemaved go Shab conlinias- Valued decision arimbuks canbe incorporated Ynts deanned dour. & Boraine Messunes fon seletingy Aitidy ~7 ONE WAY 4v avoid This diBhcurty is + Select decision atinibules ased on Same measure othr han Wibornation gar. on allanal seasune hed has been Usd Sucrsfully is the goin orale . 7 fu gain aatio meus penalites aHribudar such as Dati by iNconpeakng ato Called Split iasmahon, Spliintanmabon C509 = ~ 21 is (gy i | => tha gain Fakio measune 8 edn Intoms t-te canhen Gain meatune, as welt at this sobitiesmaken ay fallout Gain C5,AD [srntakon 6) = Si tnwomahon AD 4. Handa. Tnaining Examples with Missing AipiibuX values « ee es a 5 Sin contain Cage , tht available dal, may be missing valu don some ation buks. Fon examplt , 10 a medical domain in which we wish porechel pakiend auecome basil an venous Aabsnatiny 7elts, imag be that tha dab tes? Blood Te} Keult 1s avarlabe only ary the patents. a ee a7 me Ju eftimads Ph missing caavmbUK Valle (eaa based en other examples fan oh) This cédnibule has a Kren Valu . 5 Handling Avorbales with oiddonng Cass, oo) mM same earning thes QMoibuks very significantly in Ane casts both in toms & monetary cos and cost tv patie Comat 2) such dasks , We would paefen decision Anes That Wut ovo ~ COS cAiairdures Wheu fossvbk ,delying a high-cost Adal ouics only when netded 4p Produce deliabk. classifiadien 3. |r— a unt-i 2 Asiificva Neunal Newonk + An Amtficial neural Nelwonk consist 0B uni, Connechons ard weighs . Fnpuds and duedpuls atu Mumesnc. Neunal Ndwont ‘- _. . Neunol Ndwont : > ntuna nelwonx a Sykm Composed a many simple paroussing elenun s openahing in pasallel whose function 18 determined by Newonk Stnuctune , conneckon Soength.and the porocessing pod- onmd al compuking elements on nodes. 7 Neunal nelwonk —Jeooning methads poovick a sobust appeact] to approximating neal -valued , discorele - valued and vechn-valued Aanget funckons . ; +? Fon Cerlain types db pooblems such as Learning te intonpnet Complex seal- wonld sengon dada, aatrficiad neund netwonns cou among Ah most ebdectve Neaning methods cumnently Known - Neunol Nelwonk Lepnesentatign -. Te NN Ree +> A prototypi@l exampl ob ANN slecarning is Provided by Fon Fommrleau's C193) Sys ALVINN CAutonamous land Vehicle ina Neunal Nelwonk). which is used do q Seamed ANN h SHece; an auferomsus vehielt doaving at renmal sperels on fubhe hiphauyr Le tu input a neural Aefoamnk is @ Zoxe2 amido} pinel imensi ties oblained frm a forwcaoel ~ pointed camera maunitd 07 thay vehichs, PT nelwor culpul is -the dinechon in which te vehick ts Heed. the ANN is dnained to minic The absenved Sleesing commands of @ human doviving “I Yehick fan appowamalely S minutes, > ALVINN has Use His leaoined nelwanks 40 succesfully dive at Speeds opt “7a miles pen houn and don alstanas de % aniesON Public highways . Shoop et sAnagh) Ahead : 1 a Shan Right Bo oulput Units 3oxy2 Senson Spur Retina, The ALVINN Sgthen USO Backpropogatign to Jenn te Stet an Ouronomeus vehick daiving ak Speeds upt 70 miles pen hou: +? thi diagnam-on th Aébt shaws how tre image dba torwond - mound camera is mapped 966 neumal network inpub, which oon fed finwand to 4 hidden uni, connecld 40 30 oudpet units . =? Neluonk oulpuls encodt the commanded steening cineckon « fy fiqun on thy Sught Shows cneight values fon oru db: the hiddn unis in Anis Nelwonk. y Ty Zoxar. toeighis inh-Ihu hiddi nibs aru chsplayd to thi Jorge maazia with Wweghh whik blocks indicaking fasihve and black indice negative weighs.@ = > the weights from this hidden unit ty Thu 30 output Unity ru dupickd ‘by Ih smaller sneclangulan block dineclly Absve he Jonge block. As an be San foam These ouipd weights, ackwakon dh this ponkculan hidden unils encaunoges 0 jum Avwond th Jet [> the Gackpnojogahion alganithe is Thy mast commonly ued ANN Aewoning Agchnique. 34 8 appropatali fon powllems cyith x allaang, chapacterishis: ener 1 Tnslanus_omt supnesenied by many artarioule Va PAS: ee eae 6 delingd ovr insianas had can be descaiibed by a vechn db featunes, such as the Prd. values 1 IK ALVIN examplt. jos 0m a veclin GF Several ea) 93 discarele walled aH NU: —> fon example , in tru ALVIN, Sym hx oulpud 1s a Yechy ds 30 attniaks , each coomesponding ra srewmondakion aegading du stewing dineckon 3. Tpaining exomples May geniain exmrens. 3 ANN Jeoning methods ane quilt nobus} 13 Nose In Tht “Anaining doa. 4 long trmining hmes se acaplablt. 7 Nd wonk ‘Anaining algenrthms Aypically nequine Jonge, Anaining times Than, say, decision ‘|net easing Algo. 5 Fag cvaluchon dt denned “eoget Yunchon may be HeqUANed, 3 fivnough Awn Seanning times ane sielabvely Jong, evaluakin 4h Seanned nelwone in ondey to apply i ty O subse quer inslana., is typically very fast -S The ability ly humans to Undersiond “tu Jetrned anger unchin igre! imposvtany . > Thu weights Jeanned by neunal nebwanrs ane abkn difficult fon humans fp inten poet Pen coptorons. HS One “ye dt ANN system 1S based on aunit called 4 Peniepinen > ihe Penepinan isa feed -fonwand nelwonk with one Output Neuzon that leans a sepenaking hyperplane. ina Pattern space. > Th Percepinon Sefenobs tineanly set ok Pattens. >A Poxeplnen Jakes a vechn db aual-valued inpuls, Calculates a Uineas, combination 6& these inpuls , Hu at TARE Sesult is gneoden than gome jhaesheld and =) Chow sc. > Mose pareciselry 1 give inpuls a, Fhoough rn the Oud put Oh, -+-%n) computed by -thu Perception is LV Wap + Wy +WAQ+--- +H ZO 0 Ob, -+-- An) = “1 othowist. cuhene each O32, 18 Q geal-valued constani on weig hd ‘Yhod delenmmines thu combs bution G wPpud % Jo Whe Penceplnen curpul -@ Representational Power dr Powephnons .- > We can view “thi porapinen as orepneserting a hypenplance Aecisin sunfac in ht n-dimensional Spaced insianas tthe Pence ping ourpuis a 4 fon inglances Aying on ont side & ahs hyperplane and owpuis a ~A -bon insianas eying on Ihe othe side. }> The c2uation fon this decision hyper planu is BZ zo. }> Some Set de posthye and negahve examples cant be Seperated by any hypeplant. Those ‘Thad can be Sepernaldd ane called dineanly sepoabk sds ob examples. = + A sea de Amaining example £ ee mining oe The decision sunfau a Porepinon Seperati . Wee nod classifies tum comnectty. DHEA, Are tht pereplnon inpuls. PoSlive examples One indialed by “+” negahive bye A single Peaceplnon Gn be usecl ts Stepnesent mans Boo dunckens . foo etample, ik we Assume Boolean values d& SCoiue) and -4 Pals thm one way to ue o two - input voeplnan To implement tha AND unckon 1s do Sel thy weight. 7b two closes d+ Folens can be sepenaled by a decision boundany , neporeserled by Tht dinca, eruation tun Mey Gow Said to be dineanky Sepenarle. the Simple nelwonk car) comnecity classify ony Fala,> Decision boundony ds Jinganly sepenadle classy can BC Aedormined either bY Some tanning procedures 0” Fy] Solving, Jinean, equakon syslms based on repose adive Patioons db each classes . > Tk such a decison boundary dacs not enish thin the two classes ane Said to be Sineasly inseponable . Lingooly inseponable powblems can't be Salvedl by Th simple nekwonk , mone sophisticakd anchitedune 1S nerelea}. > The Poreplnon Toraining Rule :- — s -7 One way to Jeon an acceptable woeighh vectun is to loegin with anandem tweighis, ud Herdively apply the Pencepinon 4o each 4naining example , modifying he Pencepinon weights whenever it misclagsifies an example. > This proass 1s depeakd , Herod though the Jnaining examples” aS many times as needled unk) The pencepinen classifies al dnaining examples as many ames ag netde until Shu Pencoplno “classifies all maining examples conmectly 4 weghis @u modified ad each skp acconding to He Perce pinon maining out, Which davies Th weight (x associakd wih ‘inpud 1y acconding to ih dult. WEW, +AU whou Aw = VUt-o) x; Here +15 the Jeogel Culp) fon dre cunnend -lnaini example, 0 18 She Oud pul genenakd oy the Pence phrun| and YL 15. Posiive constant called The earning abs -3. Gaodient Deseent and th Delta Rule - — Altnough the percepinon sulk finds a Suastfel oveight Veet twohen te training examples aru sintooily seponal 34 can fait do caves Convenge 1 The examples ayy not Lintool Seponrable. : i A Second Amaining guile catled tru deta sult, 1S desig ne to overcome this dtBicully, TTB AY Anoining exampks ari nei dingonly sepenable, the deta cule Cnverges ‘word a besi-Qi+ apRnoximakion to The tong concep} . p> The Key idea behind th delda ouule ig to use goadient descent ty Search die typothesis space ok possible weigh vectors to find the weights that best {ih the Anaining examples. 7 Ths Tuk is imPoolanl because gradient descent provid ‘he basis fo the Sack poropogation alganithm , which Con Leonn networks osith Many inlencomected Unis. > 14'S also important because Gradient descent can Serue OSM Dosis fon Leaning algoaithms that must Seanch “Inswugh Hypothesis spaces comdaining MANY Ai ener) Tyres ok conknuously. parameterized hypolheses - > fu delta Amaining cule is best undenstoud by Gnsidening Ane Jos ot Tmaining an untnsvesholded porceplnon, ie. dingon emit Gon whieh ht oulpid a is given by OMB : Thus ,a diner unit connesponds do Ihe isis) stage oba Pencepiasn , ayrthol the ‘Ihaeshold. > In ander to derive @ weight Learning ont -box dineosr lu unt, Jel us begin by spec's wing @ Measuau don “ttyAoaining exnon db a hypothesrs Coerght veclund, oreladive doth Anan examples . : : b> Although ioe ane mony ways to débine This error, Ore common measuae “Mned eit) Tun out to be especially — convenien) 1S nig el S Cd-os)” whane D is In sd ob “lowaining examples, 44 $8 the ‘lenge oul Pud for Anaining exampres d, and cy is th Ourpud G +ru dines unit fon cinaining example a. By Anis debinakion ECG) 1s simply halk Hu S2uared Adberence between te Forge ourpd t4 and tty ey Lav Output Qg summed Over all dnaining exampks, 4a V'sualizing “nu HyPotnesis spac. > 310 understand su gnadient deseem alganithm, i+ 1S helpful ty vigualiae tha entine hypothesis space d); possib) cuerght vectons and then associad € values, Hens the ONES Wo and Cy, Hepsesenl possible values fon thu two cucights dk a simple dtinean uni. thy tos,w, plane thenedan. O r t HepHesens the ening hypothesis space. thu vankod Onis indicates -the emmon E oielahve to Some bined Sa & Anaining exampleg. The comer surface Shown inthe sigue ‘Thus Summey ‘We desinability & evoy cexight vecloy In-Ihe hypcthe Space. > Giver) thy Way in Hridhich we chose to debine €, on Jineas unity this exson Surface Must attuays re parads witha single Global mintenum ©. Denvahion o& tu Gmadient Descent Rule: ee CT Racent Descent Kult, > How Can we Caleulat tre dinection o& Sleepest descent dlong tt. ernon surfoce? THis direction an be found by computing The deivahve d& E with nespect to each comPonunt dhe Ueehn OF. This vecta cenivahve iS @bltal Ane goodie db E with gespet lo B. andkn AGL) Veh) = [9E 2E ze | dw. 5 wn Hp Noe VEC) is sdk aveca, whox comporums cou Faakally enivatives o& E twith nespect to cach ob thi wy. L> wun imopnekd 03 a vect in coeight space, “hu goadient Specifies “ta dineckon that praduas dh Seepest inc ease ine. th negakve drthis vechy bhenefanw gives the disiechs Ob Scopes decocss. 7 Sine the goradhent specifies th dineckon ok slopes} inccreat & Ed ‘Vsaining giuk or qaaclend clescend is Bo vpLaT wheu Als = -Yv EH) eee Now Subsivtuk Equaton @ inh Equation @ yields “Shu weight update sul foo gradient dlescens [ Aur = VS ed-Oa) 7g © Shoehostic Approaimation ty Grdiend descent» meee eee Gradient descent is an important gonna) paradigm -om Jeonning, 3+ iS @ Sinakgy fon Seanchiing “ihsough @ ange@ i. Om imlenbace infinite hypothesis space That Can be apphed Whenever W the hypothesis space contains conkniously Poamelontel Mypotheses and 0) the ean canoe clidboun hated avith oespect to these hypatheséy panamedars. tu Key Prackeod abbiculties “in applying qpodient ant w Convenging toa oad minimum Gn Somehms be quik Sbw and @ ie then one mulkple docal minima int enon Sungace thin thou ig M0 guananter “har tu proadune will find thx global minimum, Goad ient— Desens Goining. examples, 1) =o #€och tnaining example is a Poin othe -fonm (22), whene 3 is the vecton db inpud values and t From “tit input nodes dala gees Fhoough jhe hidden nodes Ci any) and tyty oulpt nodes. Thene an ro cyelt3 as oops instru nelwon. > Two dikkenent classes Newank anch lechtuses 1. Single-Jayen Fes! Fonwand > Neunans Du decognized 2. mull -layor Fecal Forward» In acyclic layers.4 Single -layon Feed Fousoond - One eee > tu simplest Kind gh Meunal ndwonk iS single -laycn Porapinon nelwsnk, which consist db a Single Layer db oulivd Nodes. iu inpuls are fed dineclty tote outputs Via O Seovieg o weights. Thy sum Atha Products 6 coeighb and inpul 18 Caltulaled in each node and ithe value is above Some haeshold the neuson fines and takes thu ackvated value, Othowise i#4akes the deockvaled Value Input aye Gutpud ayn & Oo Sdunce nodes neusians > 2 TrutkLayen Feed Foowand nelwank - ———— > A mulk layer Feed —doowind Yeund Newark ig Q Network consisting && multiple ayers && units: all d which aru adaphve - > Tu nekwone is not allowed 4o have cycles 4nom date, doyens , hence tu name “del tonwand”, > Lal us consider @ nekwonk witha single complele hidden Voyen. i-e. the newonk consisis db Some input Nodes, Some Oudput Nodes and a 3d do hidden nodes. 7 Evers hiddin node dans inputs -fnom each @& the input Nodeo and feeds int each thu ouput nodes. > This Shucluxe is called mulslayo because i has a Joyo 6 Processing units i-e hidden dayaer units 9 addihen tothe aw put ants= tiddin layon mul, layer Fead Fonuund Neunal Newonk , > These Newonns anu called fea Gonwond becousc a oukput fe om Jaen & neusons feds forward intr near dazu dir meus Thom oor Never any backwand conneckon and connechions Nevcn Skip Q Jaxon, > Each connechon between nodes has a tueight associakd suit] MW. Jn addikon the isa special tweight Coalled tu») ‘that feds Into evey Node as thu hidden ayen and a special coeight (called 2c) hod feds into evoy node althi oulpud Ho These weighs aon calls thu tras and setts tot | Values fon the nodes. Snikally , aU db the weights ane Set to Some Small nandom values neven 2u0. > Every vode in thu hidden ayer and in the oud pul dayen Prous iS weighted input to pooduce aN outPpur. This Gn done shghtly ditberentty ok Aiu hiddin Layer , compagud 1 te oulped Joyen, { Input _units > ihe input date fou provid Your Network Comes “Ihaough hu inpud Units. NO procasing axes place In an inpuy unit, i Simply feeds dda into -ty System.2-thddin Um} '- The connechons coming oud dk an inpak unt ne Mens associ 1 i fo hdd have weights associated curly Them A weight going i Unit Zh foiem mpud 4; Would be Jabelecl Wh, Th bias impur Node Cx 15 connected to athe hiddin units, wiih weighiseohg , €ach hidden node calculates ke weighted Sum ds inpuls and applies a Ahmeshslding function do delemine the oulpus Ob thy hidden ode. P? ethneshdlding incon applied ab-the hidden nod 15 ‘ypiaally Chinen a skp tmchion on a Samed finchon. 0d 22 wn, 4). ny “ SG) = z 8) ~ ot The Sigmoid Thneshold nit Petike the perceptron tha sigmoid unit finst computes a dinean combinakion ds i$ inpuis, ther applies a Ihaeshold tothe desu}. Intnis cose db thu sigmod unit, Nowever, -Yu Ahaeshold oubpur SG Conkinious funchon dt 1S Ynpud- i > o is ohn called the sigmaid fanckon an altenatively thi lugist function. Note #2 output anges between @ and 4, INeaeasing Monotnically wih AS MPU. Because WH MPS a voy doog input domarn toa man dang a ouspuls, % is OBO MAKE tv os tu squashing duncion o& dtu uni. 3 QuiRal Unt Funchonally ‘gust dike tu biddin Jayers. OUlpuls ane Passed on tv The wonld oudsids the neural nelwonk,Tne Backpsispogation alganithm ’- Be Seeley alle Each -Inaining example i 0 pain & fom (7,2) When ® ig the Veco Sb neduoane inpud Values, and ig tu veckon df }erget Newark ° Output il TL isthe Leaomning aut (¢9,05)- Min istanumben te Mewonk inyus, Mhadin he number Gb unis ny hidden Joyer and nog Fe number s oudpuda Un “The input {nom unis \ inh unit 3 is denoked 25) and “he weight fom uni Yo unit) is denokd oj}. Creo 0 Fead—dymwand nehwons with Vin INPUlS) Mnisdin Hidden units Qnd_Noyj Gudpur_units ’ Lnbhalize OW ndwane weights to Small random numbers (4: ‘cdwern = 105 Gnd 05), ¥ Oot! the denminahon condition is me, Do fon. each (2.7) mM Inalning examples, Do Poropogal “tha input ‘fonwand Abmough hu Nehwone - 1. Tnput the inslana Xdothe newone and Compute the * Qutpul Oy ob evry unit in thu network Poropoga thr exmens backwand Ahowugh nebwons 2. Fon ech Newonk output unit K, calculate Ws enn derm Sx ian, 6 p< Ox 1-0) (A= ON) 3. Fon each hidden unis h, caltulal 3 enon len 6 Sn< On C\- On) SZ agiirhe 4. Updali each nelwons weight Wy; Wojp Wii + AW “when [Aen 22 5 ir Sicashe gnadient descent version dy" Back pnopogadior algonahm to fecdforaand Telos Contaming two dayons ob sigmoid tunis .a * = The Gackpoopogation algonithm deans Ih. weights fon a mulk- Joyo ndwonk, given a newark with a fired sedis units and IMorconneckons . > TV employs gaadient descent to adtemp) fo minimize thu Suaned ermon belween The nedwonk ouput values and he Aasaget Values fon hese oudpuis . [7 The Jleonining pooislem ford by Bacrpowpeqation is t> Search a ooae hypothesis spac. détined by all possible weight values to2 ak Ae units in the Hewonk, > fee One méjon AiBbeena inthe cos db Multilayer newonks iS thot the simon Sushaa con have mulhple dom minima, in coninast sto the Single- Minimum PPnabolic ennon Sunface- > Undomunadely ,-this means that qnadient descent is guananted only to Covenoge Fowond Some Jocal minimum and not recessa- nly The global minimum ennon. =) Despile this obsiacle «in pmacka Backpnopagation has been found to produce excellent aesulls in Many dread -cwonld applications. Deivahon & Backpropogadon Fulu - oe pk > The dewwalion athe Shasbe gnadient descent Sule iS concep) ually stnoight fonwand, but oeauises Keeping ‘track ob a numben Bs swosenipis and variables We wind follog the Notakon Show in Sigmsid -tnneshsld unit. , adding & Subseoipt } 4y denok 70 Anu 'yth unit 6 Tu network as -follovos > yi tthe th inpud te und 5 —>Ujit The weight astacialed wlth the th input fo unit j 7 MAS Zi Wy CMtu weighted gum db inpuds fos. unit 5) > 0} + The oulpui Compuled by unit 3. EFL THe Aangel owapud “for unit }- \ \? + the sigmord tunchon. \