Neuro-Dynamic Programming Dimitri P,Bertsekas and John N. Tstsittis (1996) Manaachusts Ineo of Technology Athena Scientific, Belmont, MassachusettsLearning won thought lo ot ought thon’ lary pero (Confacion Anais) Introductiona artisan — up “This book cosa sens wire deci ae mae In age The ‘utc fend con wt fly predealebt en be ated {Some exe let ert dain nda Eah dectonrr e ‘nium cnt blo ult th cot wc tae desis ‘Ector mde sn thor let cat arr uta ge, We ‘Sc lstereel i den aking peli ht minnie Ue wal est vt mer of stage Sah protien ae alleging pail becuse of ‘hota tctwves mele ad here e, By proemiog (Gr fester rove « mates emaantion ths eel ‘General DP fotos we tare ecole pen 7 taps who sate ees sng to gen tation proba tat ‘pend ons decion/enrl vIn prt me at ate aed we ‘Sse cml we moet tate j with gen protay pyle). ‘Toe Sond dpeon he ne ode ae och wale cet el a le fic ye), Sly Tecanpriag, bower, the teen eaogh oon ‘the maga the eo (i) we mt ae nto uct he “lira ofthe net ate) Wet neodaway ton orate sates ‘Thee done yng he opine (vera ering sng) ring from sae, hich etd by J") nod are oa he tial ‘lvoe ate Thee ogo am besos ome fre Stina eatin nian) +0) ha}, bral sere jl tate hag tan Bl |] dente expec ae repost jee and. Generally, euch nate opal {ova conti! hat stan the minim above Tas cota ae ‘aie tnd on th sn of he epeted eat of npr prod td the opti expected nt ofl aban pes pre_—_l eas igure 1 Stee dine ne ct Sec M1 Catton Arron Dynal Propanning 2 “he objctve of DP ocean he opti eat to futon Jn" enptaton an neo ble the a ‘em sar opeting Aa opal ply, tt an inal bie of Wreschis stomped ether smn wth Jo nel tine by ‘ing noni ands of Balm’ oqo ewe em meer, th Sr many important bles the computational eae ‘nets of DP re erwin. bras hermits and eon ery lage (elma “earn of meson”). In uch ostions opine wltion eid (COST-70.GO APPROXIMATIONS IN DYNAMIC PROGRAMMING {In this took, we ray ot opal tod hat ene around {he elton sa apetinaton of he optinal atop anton 2, ‘my throngh th of trl nrc and/or slat. prs ‘ih, weve the opti ese-tong 1°) with asta apes ion SG, where rv wer of parameters ane we at at the (cntoptonl cots phat stale te msi a he (appreciate) ‘ohn ade of Belin equate that He «aang oi.) HJ) 6 ‘The function J wilt ale the sariny faction arsine east {9 fnction andthe nue J.) wl be ele he sare ot aprosinae ‘Seog ate fo Fay 12). The gow mo J on ‘ch that once he preter eter Ed, the ation 63) Sy sat ay spe Salo) | reprosnatonnan [REPRE We ar ftrrte npr wth ye nunber of states and in scsing stone T thts be devel nth lively fo ars (2 ‘serra males). Sing loa ising fw parte4 anodction Chap wl be alle campct presentations, whe the tbular rpm of ‘il be eal the tap le repeation, faa kp able pce tate, the valce JG) for al ats} ae teed in ale In pad onpat repression ony cir Se grow era Se Tae ca eH) ween IRrepome tote oa tne? insted veer of rp ot Sacer th eral network: Jj) oto esi ‘dseptin of theta interme Nuestra then rd ‘ever of late sgt features: Naar e woud fo oe artical w hat J) approinates me (Ts ‘eng the searing ection JG) nest eoplamtary lowes (i) dein onthe eer reo he non Tr, ad) ca fut the parser vector rap ao mn ote wane the er Tree the faton JC) and Cyr "We note tha i some proba te elution of he expen Blot) + JOM Las foro, my be to compa ton nea ing dc ‘asin ea nacre the ene J re ngy eat Thav a ‘Smber of wap to dal with ec (ne Seco 1) An por {eat pouty to approximate the een mined a Baas lb) = Botha) +24) Is. Wiehe kaome a the Q fcr someroding to i) I pata, we Sin pce Qu) wih a sual opprnasnn Qe whee ba ‘car of prunes. Wo ha the at wae th opin) ete ‘Seni the approntaate cer cmepaing to BO) = aemegiinr ‘Moc wht willbe id abot the sptxiaton of th epi est {oro ai spa othe apprsimatin of Qrtae nfc ml e Ine ht the Qc ca be ewe moti sate ed rote We ths fr pinay ou appoint th pind ete ‘Approximation of he optima costo go ar een in he ps In avurny of DP ome, Chas paying pret epee eee Ing ramp A ay st nthe porn sponte eet ‘oak dite se posts al Yost ch far a moet ‘eri he psn wth het a Th oan ears ‘rca! ale to oh pion aos bertae mae hat Salar nih fr he vain etre th pon ara nae ie milly, Lng say, nde fc von Fg. 1.8. Tho the po {ce alton eoreonds te cng unten Jj) above whe ‘ris the ture corpo tothe pare vt. Vly se ‘ou sructre sted fr the onton saa i age 2 {Bhat has ered over many Jer, bse or experitain sd ‘Ban tnowelg abu an) atd theorist sy ‘eer (tne celts dpm apn Pg Tl ‘ements that iio othe felted pein econ te och of the mca f eh pops ean bate to he we ‘fmlinore olan, which as brie da aes (hee eens fae evar sate te at a Se ea ac Pir 1. St ot ping er pa -Astheches propa prin spp tion both pie, rit dia nd ore impr aged concn a-nge appacinatnas in DP. Hower, ee mporat 1 appbmen heats and inst with more tema chal tht ae bey ‘ppc and ean much ws posible fhe nonbeaite tocar, fbb Ths book wl fa on seer fart to deep thn ‘log foundation fora anal ppc To complex chase on robes wht conbies yeni rograrning,fncon positon, te snaton 2 APPROXIMATION ARCHITECTURES ‘A import ela ftionsppracination i the ston of om ‘her, that he Go 2 pret of nts Sr) oe. Intron Chap. Ge) tm he pt a, Oe poe 2 ne ‘ait iter stame type. We Soul paso be that {Eto we th tern arr ttwe” a ey oad St, ee {yu synonym fo Saye reitetra” pate, Wed tet rose ours ote dal sles prepo trate wih ‘pout colinnuite Any pe of nial appenintro olinot ‘Roving cl be od our at The nature of the pyoninting ‘Steines eft pen ou dian, and eal nei, fo xan, ‘ada afte, edn penile, apestin ee “Cao go pinmasen con on be gia ean Ue the wef foe tration proc hat ap the tne into se ve {or Jello th tr sco ode wth ature vcr ‘Sve qua tue wer may ole fe ac quan hve ied Inter tat pei wht the une “aly py.” "adraey ‘yo eed fl” Tama eae, anc complement ition toe the ih aan fr th problema bard. tare vectors erly wal when they cn expan the “omit seen” the optimal cco g fenton J By tht ‘ween thn J") ce agente el by wey sac” Fann Jf (0) this bgp eanpl thong xchange of ‘Sin fom rate often, J bcos (bat) ear oo rr [el atc of there, When a are trea be es {o have this property; spp to se apponinaton rca ‘ore fnsues ed tiny ange) oer networks ae wed ogee. Tesla, to lats bpp tom ate vstor whch she aed ac Input eral etor at roduc th oe of teat =F). ‘More peavey ott oth he mate and hn fear ete ae ‘ovo a puts 1 oe earl tak (ow the ssl agra i i i SIMULATION AND TRAINING ‘Sour mot ctl pin nmr inn elon" thon appoint sar eter tod a les ave none ring data 2m pai (9), wich represen tae oft ping F that apni. "i porto nts that trast with tun neural network sesh the DP conte thre renal sing et ‘Epu-stat pal (0) nena be lo approinate J ich at square ft "The al pny ito ete (acy or appro ‘mae Uy simuin the eo-tng cin ef en (abla pal ‘Stand ty telly improve thee posse he sialon ‘emesis cote saya! anemia il a do ot are in scl nerd ntwek ising tenes Indo thee of ‘Slaton exalt apni the option ct g uction 8 ym ta ings he ology fh bask em uae ‘ypacnation metol n DP. ‘Solan fos another mirada: tallow ee method of his bok to be ied fortes tat end to model but eyo Simul, pbs whee comeait ep model ot wala tate ssn ol be re, ter rag safer stn ot ‘Sitoprsin eine, For sh pbls, tetas DP cise ‘Se lnppiab od einen of ane pobabii oem ‘otal watonate mo! een camberme cr pene “Thee tid pote stage of sn an pithy ely the ot rant ot oct reprataieatr of the eo pyr pl tht Sarno ert oe ed ‘Sirgen, and for htc x sing nto wl ond to ‘Spoosinte ter thot Otto g rte sata, andthe wap ttl paey baie wl the wetge pro et* atti — Capt 1.4 NEURO.DYNAMIC PROGRAMMING 1h view a the renee eth DP and rl ork conepts, me the ‘ste neao-dnarcpreruming (NDP fost) to rb ely he made of th wat Inthe mal ealigens comm, where the mode eget mana nfm ering alton. Th ‘man ri inline em he deaf the teak eb ter to lars howto sake god dns ty ering thelr wn br ‘vir, and ue bln mechani fo inprvng tr rt tao ‘*llremest mechani nthe ns anopeere DP tems wel eth book, “obasng tht ow bun ost nto, and “inprovig thei actions though «sorcerer 1 late shee fo Improving the gully of wpromaton of th pt tal cts, or the ctr, oto optimal poy Tar hat bat 8 ‘dual eration ta eno! eigen cab ily Fotiatad nd iret nt of Cosel DP seme ih wee {pay trate ats boo, we attempt oly sume pecs of he cnet NDP ‘ache, we tpt wone ew sgtns preach we We Uy seme oper guns. Dept th rattan NDF, tn hen of ‘he sj nao egg to asap, sod th crepe rar scien ceafsng Yor, te hoe bes may separ ce ‘ih robles too lage sd comps to be trated a any the may A stay inpeive crest gen notated magn ach ‘atthe devpeat of bgunton pang rogram Tee (ee Seta 6). Hare nur eter sted pent ie ‘tinal eso fant ofthe poof bckgance by ng iar on, thy ting the rogram ply agaist Sac Mer ling for ‘ral moat the pan ool delsed te hanes mod canon alte chs pons ths prop didnt ue and of many sage ‘elite soc ae bested pinay tote wee popely ed ‘Soocinaton ofthe opti! costo fc ‘Our own expres has brs tat NDP mths be ingreily oie robe wine tain! DP maths weld be bay op [ban eter eure methods wo feed pies fee ‘ook we ouie sme esiering apes, nd we we Sw {ional ate for stting the methodol ahd ne he a tal ‘Sof etl fo cee "Wo ot, bower, thatthe peat appatin of NDP compo ‘sony wr inten, and fen er olde nou al fe ero, Ruthemore scams sone ig tele wwe ‘operat ot ally udercod. Fate, al of a semptatin {hd exetnesatin ih direst appreachor cane dae eae Onc {he ppratiaton ose nt canbe ted opr eos ‘ar enough foe nel tne I ths conto, wt an sel intelignetaratr, reinforcement ering i te viewed 0s “onlin” mete, whereby the cnt tog appa pve heaters pnt nl Une There ft hs atonal dave sot. We wl oe cstv in hls ‘oak ab we peer to fou on ppcaton toting Ire Sd epee ‘ystom. Aor of mining dats ae equi! wc tems Ths ace ‘hen enact be talon! nwt value te ston opereig, ‘enf teenth eremponding prosaing ogeesens we oes Se fret we al te ‘Mi nly tion a slaraaveapponh to NDP, known a op proimatin in ply spe, ich, however, wil ster te Sook this apench, i lse fan nr opti pot, ek fora optinal pry within sone retical cleat pase oy ‘ewer vrata low dimension. In pares, ce pl ‘les of en fos le) We then sume oer» to expected [E|J8 of] whee the expectation wth ep to ne rable pak ‘ily dstebtona he hal sate Ths eppronch ape to empl robs where te no ext mata! fr te tn aod the, ‘Slog sth ear evening to sgn pay cn bo ced fy Soliton, Fethemae, hight aed spay ean somos be ed fet single and toi partition af he poses On te she to, there are may poems wher ch parsettios te ot xl tld, Prterne tn wininiatin of [00] an be ey eal eas the eto the est with rept to» may ot be aly fala thd on rie et wae (oc etre) ma td they ten ore fay se fancies ewan on toe “he ger epantona lan fhe toa eo it dee sme ssuia eckroun materia DP don Stemi a Sechasie leat apintatie grins (Ca 2-4, and sant dewey the an ‘ote toc f NDP in Ch and 6. Varos enon fe ‘thology ae dice in Ch 7 Fly, we rent exe tan Che Many ofthe feet extemal to cena Se fens thou the NDP thay ftom compte ir sch ten, To Keepin pcos acho wel eases ashe the mime fata iste a he mse of sae ent ch te aly fe, Tas consent we the caption etn rike bok NOTES AND souRcES 1.1. The eine cor hit can be tnd to he ely wok on DP ty Beta, who wed thet apptioe ve pte"Infection Chap. to the was by Shanon Shas on camper hes ad by Samus {Sani Sant oe cnpatr ceca a ety Ba Sete, kee SAE ois {ie pyc, by Sin Beth) on tempore rence thd, Stig Waite [toe @arsing nated tbe moder deep ‘mnt whch tng ter the ar of fon appessmatin, ‘oulsons sad DP Ts wok of Taro (ea. Feo, (et Tchgnnmon we he fit to dames impr cso ty see and changing cio» ch rer flre {ie bemia srt. "Te extent suey by Baa Bal sd Sg S09) eros by Weras [Wer era) and ‘ther pre the eed wl y Whe and See WASH! pla nthe conte bean th aia tligene/ tore Tain ews sate otal twry/DP iwi acd ve ‘Say renee DP tntbosk by Bera Beri] Sere ‘eval weit fesbopial conte! ott, nag ee oft [NDP opronss tht raed nth per ph pet Pip wie in hi gro ok te wives, ‘Bathe bt ormat alent ‘lone fat ars spond he ngage nt rade ter sch fw conga Ia rten inde ngage of mabonatis (atie) Dynamic Programming