INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor INTRODUCTION When the Symbols one gereated by the Source they do mot have eqpal probabilities alecage Fa example Occurence of letter 9! 4 English language va Companed wo that 4 len Tall He symbole genaated by te Aounce one allotted same umber 4 binany digits | then informacion Cesrying Capacity a binary digits UTM not be efpiecent ly ubiliged tn case 4 lew qepertly oceusing Aymbols . Source coding Savolves efident means dey encoding the Source outputs » These encoding algnitins exphit the stalisheal properties 4 ste ounce - The frjamalioo casging capacity of the discrete channel elepends upon Vesious FranaiHion probabilities acs dhe channel - Sy study the therte channel ove a ekiwmine the amount a Snewnation lost acrow and channel noke gf Pnjamadion jransm MionINFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor 1. Encoprng of Tue Source output Sounce Encoding Ss the proces by which the output A an ‘nmmaden Source 1S Converted Ynto a binany fexience The quscken bleex that pexjamns his tas fn a cemrounicakion Aggtero 4s called the Source Encoder The fnpuk to the Source encodex ts the Aymbel Aeapence exited by the finfamedion Bounce - The Sounce encoder omiqns Vealable length binary code wads to blocus 4 Ayobole and produces & binany Aequence ab tke oukpuk en bleene 4 Ni. Aymbels fo He Encoder opaales » ome *t Lit sake an opkimucn bey 4E will produce an avenge Gy bike | gymbel | here a Ss pli) log, Pom As the block length N Ss fneveosed the overage oukpuk bit mote pe ayebel nay cles oy) i lurniking cate at N30 the bit make por Byrab Sill apprach Gy and Gay vi) apprach A Thus, sith a lange blecn sige dhe oubpe “HS ‘inewnakion Bomce cay be encoded ‘nbe a binary Aapence Wh an average bet note appeacding R, athe Bounce ‘imjamnation rake . Dept of ECE,SJCIT i <) bits gpntel aie N Ta)INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor % The perforoance af the Encoder fe usually measuied ‘n tems the coding etficeny thek ta defined an she aalio q the ounce infamatfon aake and tHe arenage oukpub Lik acke q the encoder % Coding effidency should be as high a+ powible : ddeally iF should oppoach onity « A There ate mag alyetros available gor elengring a gounce Encoder. “the most poreyul , Biene le Aouace coding al yrithen aver by Shannen Known Sey Encoding Algoithn e 2. SWANNON'S ENCODING ALGORLTHN SHANNON 5 ALGORITHM) es ‘ i Hyon x Ths a simple yeh. poomdel alge coder « design ar chve Source 6m “ be qammalated Aousce encoder an * The dusign a te as dpllows : i me a conatels % ‘op encod en > Th 4 ¥ post syrbol eae! ble menages exch menage containing Dept of ECE,SJCIT aINFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor Se He probabilities of thee We need + code 1 aeplace ste “4 memages ie code wd “ih monoge mm 4 length ny bike) phere nj using a anigne binary ia an fo agen % objectve is de dind wy ards # tata cy she syrabel such tak the overeg® puabe oe pot #4 Be wed fa coding pcheme is Tn other wade e 4 a eal. Py leg, Bt be 2s. aheve Several algnitims with polukion jo he : : Ignite gives 7 problem have been prpecd | athe alg 4g ee ane em me v 1. Suppte dhe ay menage! 4 ile puch te} aoranged in der 4 decreasing probeab 4 Pepe Re * : ; eh 1 op tee Sab f = os ke oe F,20 “et inkagen auch thot ‘ log. ea) an < V4 log, (&) ie ere binary, “Then, the code le te a the pretion Fi up 7 ’ expansion 4 Dept of ECE,SJCIT 7aINFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor Bounle & The algrithen qiven by Shannen yields = encoding procedune dhat hase the dolleving properties: 1 Memages of high probability oe nepratented by hE Code wade and there of low probability ane neperented by long code wade. 2. The code nad fx me will Arypen per all cc? ee powible te decode manages anignely pom ah Code wards. ely The wake ficiency © ap He encoder using ON symbole fe degined ~ [Bw .LLAN Te GQuatity kKraer Ineguatity [ keart - MeM e and Aufficient Condition a weg A > B& necew an decodable code to be instantancous See ee a ee other code - wand. onetantanesus >The condition dn extstence g an hee Code wih nad Leng hha bo --Y A ¥ cee 2 7 te1 oe here Te . num i Hoenent abo! use: a. -mamben Atmel 4 alphabet % - Dept of ECE,SJCIT =INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor 2; - Wed lengt, in binany af gite the code - vd] cAnespond ing, ee Ayn Y - Number oq Bounce Agmbels > stipe fw shannen'£ encoding algpitien as qellou : decreatin 1. Lisk the Sounce Aymbole the aicles q ening Probabikbes 2. Compute athe Acyrences F,=0 ee 2 a Pa foe ee 4 ¢ Cope Efficrency AND REDUNDANCY £2 — — code i ven > The average length Lo the rt 4 Los Boonies beh] yb ies Whae Py > Pi, Pe, Ce pobabiliber 4 bq oped | ufy PR ee the Aounce symbels Sy, wad mnbo le mi mae <7 NY be the aepective ‘ ms, lengthe in bike tHe bode onde + By engthe in bite Hh d Sy Se Dept of ECE,SJCIT 7Dr. S BHARGAVI, Professor INFORMATION THEORY & CODING (10EC55) A wes) = By Pkg a bit | ymbel that oad tk ts obsewed Lops) Fa bina codex Le tken fn moray codes a (0) aleyned a ce prs) | ie a > The coding epcieney a 4d by 4 4 Sts) | je once 2 e c odes fo my x Le + Mae ut Alisogs 4, 21007. b “he ceding redundaney dented *G Dept of ECE,SJCIT 78INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor Equation © Sigrities hak Langer the Value + Aymbel probability gmaller vil be value 4 ‘ni’ and a Jeon encoding digits will be Buffierent > 4, ny heppene te be + fraction it i# rounded - 44 to nenk integer we by Mane) <0ie tmx Ca) This be sendtten oF Can leg, (=) leg, (er) tea leg Multeplying noughouk by Pr and taxing & Aummakon Ja all Vesying ben pay 4a i + < ni Pi A op: a Si Bnte(n) < dan eg ; PT ie log Dept of ECE,SJCIT 5Dr, S BHARGAVI, Professor INFORMATION THEORY & CODING (10EC55) Propiens Le Apply Sharman eneeding alanithm to athe ellening Mosag ex s1 Se. Ss Find code-nads using o-s os Om Shannm't encoding alanithmn and also dind code dae and redundancy Solutren Desig . Probabilihy eon 2 baa 1. \eg, “Fi tea, (4: ey) ems the ts). ' n, tea. 23) < log, (33) with F,=O 2 PrP, 2 Ok Fa 80 Dept of ECE,SJCITINFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor To find Binony Expansion o-Sx> o-oxe >! i] : ay ney 2 Lae se biteleymbd 3 anmsPs 2 Symp = mR aa. ted Loe Cosy en + Cos Leow bits | symbol yp + Coa) ) ) ae ss a1 38/- 4.7 Dept of ECE,SJCIT atINFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor 2. The source emite an independent Sequence of Aqobele pom an alphabet eontisking a 5 Aymbole A,B cD and E with probabilities 4 Yale le, Fied the shannon code 3h 4 Sie respectively - d@ each Agmbel and efficiency and rcduadancy | ste coding scheme - LON Sowwtte probabilities [= Fi | Binary tapansion + OOOO Symbols Shu 2 ° Ale | Sle elie 1 3) 3 ale soo! } io alte baale) lal - ilo 216 vea(q) 208 et haa “ih a Fa ph Alu, 77 Fa py 2 S\te mses Fae es elie. mand mg 23 ee Dept of ECE,SJCIT 82INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor Binany Expansion Sl = o-31as O-3)25x%2 6-4250x2 —> } 0:2 Sore —> o-So“%2d ieee oc 2 Ali = 0 S625 o-sers xu? | }.22S0x2 —> o-45eKd ame —————_ o-geh ——— Boe og >) log. >) + oe log (=) +2, hae (4) + \fz )) fae Cn ec ees ae Te Le 2-4375 vel 1 = WU 2 anger 5 ae = jemate 2.6311) Dept of ECE,SJCIT a3INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor B. Design a source encoder da the ‘npem ation Bource given in digene below . Compare ~the average oukpuk bit mote andl eicercy 4 She codes gm Neda ced s heh TRee pragranINFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor Enceder fn N=! eqeeenrtt] B= 1-44 bik] ymbd 5 Ge 21.274 bh] spate Dept of ECE,SJCIT 85INFORMATION THEORY & CODING (10EC55) Perfomance of the Average oo bits werege mening bt 2thze | Salas) - county eshiag | | T2|128 ailing Folig dali lonhas | los|izs | los iss Wifes \u4hies |. -uuoell juthes | A Hy, = > Encoder ih af 2 Vesey | o |. [elas - 00000 - ole) ello lo slooloo + leleoot + lotllole + Moootd sHelloo + Mon + Mloolo + Mlolot M080 nite 1-274 4o-S6y/. | cc.s/, | Dept of ECE,SJCIT aa i Dr. S BHARGAVI, Professor cy 000 | eel | eno Olly | lool | tele loll | Noo tono oul Fie Melo 1100 tire | BST wy 1-30 bite | spb Ge!INFORMATION THEORY & CODING (10EC55) Dr, S BHARGAVI, Professor Apply Shannon encoding Cbinany ) algaithn dx te 24 aden tatension 4 dhe Source . How much hin the code ethrcency be Smproved iS) S20 Fs s o-3 o> ° cL H(s) = t- 48s bits] sinbel fee syonbel 7 1-327. foe aa syrabele 277 mder eubension will Dept of ECE,SJCIT a7INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor Probability o.2s ols ols Dept of ECE,SJCIT 882+ SooKc€ CODING Tyiropuction polenereeend es When the Ayrbole ane gensated by the ounce they do sok have capt probabilities aloo Far Example, the oceumence . letter 9” fs lew compared sto that of English langucge lekker “a athe Source ate IE all the Bybee grecrated by igth, tea {nfaration alloted Same number 4} binang ot 53 SaDr.S.Bhargavi_Professor here 00,01, 10 ane the code- weds fer Sy $2 33 Sh aspeckvely 2. Non - sangotark copes AN block code % foid + be non Bingeles . ‘hand oaly ee ae Atskock and easily “dtabiggssbable” aoere = another - Ex: consider Ss = § ss, 4843 wih xe 40,15 us oniqn code-wnde gn each 4 these s0U2C€ are ue thorn in dable below - Saree cere Taste - 2 COnIQvely Decopaste copes A tron Singalen code ‘s said + be anignely decodable’ % “unigpely decipherable ” th every Code -w0d pistent fo long eawed AeqpenceDr.S.Bhargavi_Professor Ex: Lek us contider the Secend antension aq] code- 4 = | ends 1ooo lool polo roll 1100 riol | | tel o! S454. | at Taste -3 Definition ; A block code fe Baid +o. be = “amigpely decodable i y and only if ye este Code - wade ce geo e te nen - Bing uleot 1 evay finite velue 40° Ex: Lak the recived Aeapence ak dhe Meawiver tn the absence qq noite ee this be ocolloo > phen Qecived Aeapence % TN A Te dhe mly posible ey decoding « — Code -& te an Unignaly decodable code devadled > Tp Table -2 code we fg used, then cottee is om $2545, & 5,5,5,5, A SHS fh S,5,545,5)Dr.S.Bhargavi_Professor Since’ there is me than one -0qy decoding dhe recived Seapence | Table-2 code -@ 1# not uniquely decodable cocle . 4. Tnstantangzous Copes SN eerie eee Boece cate it de mecognite . aoe Ge Rm A aniquel. ee the ~ Snstantaneous” f end 4 any Code-ward ‘in any recived seqpence , Withouk aference te Aucceeding agree be there ok , be 3 no time delog in the preets of “_ a i when AL code-c S used then te fe deeded S, S451 > Ab code-D is used ee ete feed aus 3, 5, 53%) t, decoded af 3 AL code-& fs used ther ce pee alecoding the ee eee and then the * mbols asnive ak the meceiver they ote decoded without ae ae to the Aucceeding agrabel - Theres a *netantaneout Codes - ee te used dm decoding aod eae ee Bina be have a When codes-¢ 4P me ot When cod bo} 0 aauives Et lecoke ce robo! a then mot in a positon ato vat gn de ok the Seaver all dhe code-nade fn act an Instustancous Code - ucceeding qo avive The stearon gn this te tha code-€ stent aith ot -. Cove -€ feDr.S.Bhargavi_Professor Optinal copes Mh ee ee code” Th hes us average length & . de = Bounce with a given probabi ly assigromenh G2 dhe Bounce sgrtels : Note : Prpertion on eda oe bree di can be Prepresen aS Aaa circle cliagnan Lo Bhoum Fo ques below : Nen- slex Tr CE Bloew —C ote Nom -uniquely dlecedabt tee a oniguely decodable > Non - Tnatantaneous a > Nen-ophinal Tnstertaneous —|_, otal Eig = Tiuthoding code —prepety free diogrermDr.S.Bhargavi_Professor Prefix te eee ced, oe eo oe Leh home code + “theo fa at fem one . prefixes” o4, code x . Exanpte | ete These are the predines 4, code Test for InstantANeoos RoPERT ¢ pRerzx coe Tustautanccus peersey (PESES Feors and sufficient condition fr 2 uniquely RA necenony thot decodable code to b Sabtantancous “8 “slo complete umd a a code bem predin other code - wad” his tek con be applied do any Pts aot code fo Aekinmine hetier Th we have to choote Ayrbels Bakishyng peedim a dhe “ay” number 4 founce yrpaly , be can de only when a <7 ° Ther, theo ue hove bg fo eo echens te. Aypabole to fim instantaneous code-whds .Dr.S.Bhargavi_Professor a representa the mamber af menage encoded “ale code winde qq length 1”. Thea, ue bare fe v2 oe > fa fea fan getting on tnatantaneous code, a onl astont encoding using (r-7) agnbel yy fot digit and the second digk can be aU t 7 Agrabole 4 the Code alphabet - ‘ 4 mi) wT Fa fo 2) nm, 4 Cr ae = U4) oe Oe ane nN, $ 1 > Tn a fimiler bay he umber A a oat encoded 4o 3 digt: uhich hos te be dt o, number ovo code-wade« coverpending don, and % 4 Cena or ca) St ne of > Proceeding this bog be car aisle i Ge) ci 2) : a Aa. 7 = v7, Fa f23 na € ny ¥ va ues : ee ae F Mallitljng te ae) Atreushonl by 4,Dr.S.Bhargavi_Professor Weiking the 45) tn “i -G-1) as ) < =) hos to be % Sina dhe actual number q mesages 7 os vateger fe itte e eg Ca) Ge , aa (isis nF) + Gane — ny terme ny tame ed a ' OO ,, _ oe 1 =~ Ems met me — + S14 ds) oie ee you We Wear unite ee Gy) the Kooy FneynelilyDr.S.Bhargavi_Professor Note : anit 4, pad length ft, “bh ts a hak fam fp ” Binary Digits” Be detonate Siqeulenes sesayt - Tnegpabiy Exanple = lr ut consider 5 different coder - Bhon in dable below = £4,433 Sous Spabot | code-F | s, 0° Sa oO} Ss 10 s4 L For Cope-F 3 The Rad-lenglhs in kn SiS. Sy 454 dca bine = Doe hoe da > 3 Subskteking these values tn 4 osDr.S.Bharg CMe ee Trequality ie Baishied with the equity Align - ape The leroy Tregpality Aoys shak given the vad lengtts 1 2 binitke each Jar coding 4 mestage < tpmbele , th Te posible to conshuek a0 ee sokiobging pred prpoly avi_Professor | binits Code- G Aarafie rd Iney ally . at pavsible to conshuct an Instantaneous Code - 2 2 binits a, i eg bint Go ee 8 ef! binik AL = = a Lk 7 ae 22 Sf ee ee re fev the ' code-H Aahsfies leap Toegea — : re ia pawible wad-lengths 1,2, a6 binike | code - Constuct an ‘Instantaneous> ~ peyt q ate code wnd Jn 5, For cope-J = = > Code -t does not so + With the Lard lengths a Dr.S.Bhargavi_Professor Jy sa bint 4 code-1 Aartisfres Kraft Ioegpols ly fn ne eae ee ee eee eabinte &,-3 Linike , A, = | binit a cy a 1 ee! 8 a 3 Het the leroy treypolity dhe code vad de instantaneous be couse Sh ie a pretin G ot toe oa ae SE is not Bo ee ee eee oeDr.S.Bhargavi_Professor ProsLems Ve Which ap dhe gotloning Bete 4 Wad - lengths specified Yn fable one acceptable Jar dhe exittence gq an ‘nshaokancout code given x = foi} reir of Hades of kdad-legth A; Ldad code -P eae ej ed te Sovvutton +: x2 for ay For cope-Pp: 3 Meaye Ineprality of” is i 4 2A a ee aa os Axes teh ven 2 -3 St se eect ‘ le ee vat aaxt + 48S aA Lo+ ae ae = 0.90535 414 + The votid-leng the peated qa code -p are attephble dt conshuction q a0 ee Aackthes Koay Snead ty ince itDr.S.Bhargavi_Professor For Cope-G ~ a hi Free Mnf Treyality ao 1 3 Ot is impasible to conshuct an Inthadtneous code Wing code- Conditions because setae tnegealily is nok sakigedl : For Cope-R = aM Pay Krag Treapalily 2 + € teu -) 2 -3 Vie al 4 At beh 27 . B+ 4 re a ia posible te cemahucl an Indlankaneous code Code -R cordihene and Kerape trey lly i. alec Aabefied -Dr.S.Bhargavi_Professor Cope EFFICIENCY AND REDONDANCY oe ees AND * The ” average length ie Cle code 7 given by . be 2 Ph bisihe [manage - sqnbel --— 0 — Where Psp pe =~, Py ane Ke stespective prbabiliies oo ae agnbole Si, Sa, -77- a) ane te anpeckve 4; fe ee ee _pads wt lengths tn bintks 4 the code q te ayrte fe WK the enbepy 14 gives by v a el ---&) WC) Si eg bik | manage = et ave prbabilikes oo Fa “4 code Koaraing ae a byebsle py ond the und lengths Eo iG 3 nod athe average Nengih tia cabevlated werg. “Y the enkepy vsing ee) % ot it alocye a LeHG) ta 3 ad > Hts) fA tay codesDr.S.Bhargavi_Professor Where Hy (s) = Enhopy in Y-aay units |masage-Sipobd Hy) aoe a O57 ‘7 Se oe) ers) bite [-momage - grated + Fis the umber 9 Ayoert ape ole used {7 code-alphabet . Pa coding EYicerey at Aegired a We Bice) fa binoay codes ---- (6) e = oo ad 4, = BO) Jy Hey codes . Sine dhe Aenominaln Lif oboou i fe ye de ee Bepel te dhe mumeradr re di e ay than % The” coding Redundancy "ts depined ao bea coer ED Note 1, and Ry ore usally enpee porntages : 20Dr.S.Bhargavi_Professor Exanpte A founce having an alphabet 5= {51,52 55 54 8.4 prduces there symbole Lith aaspective pobabilitves 4 Cig eq ate 4 7 2 ws Tn \Whewe these symbole ane Coded os she Cy bind code -effieteny Sl nedundarey? able” os eode-1 OO tind coke - apc eney and redundancy 7 ccs | [Sone Ryle G) code sHicerg 1, 4 = sprlhs 1 \dad lengths fo So hed =e 4 bith Popebibhia YZ source A eobele ane Po Pat 2\Dr.S.Bhargavi_Professor 5 Le z rhs = cet +0, + Fy Me = (4) wt (4) +(Z)err GO +(4) \ Eobepy HU) = S Py leg + eet 0, + Pe be eu 2 . 144553 bF ee dae 2 Ws) The code eficierey a ae = 1.94583 nto /- albstahday he Code Redundancy kyDr.S.Bhargavi_Professor (i) he wOdelengthe one t= hedge 2 binthy ond byeles & binits ee z ph (¢) tye (alee (z)e+ (so &)@ a Meee) Lists | enenege - apebel teh Llegt * lege +t tga hg : - bymnbed Wide 1 14583 bits | menage 4 ie Code officierey 1. * ws) L Sut acetate ean oe 2-16667 Seat - rare Companing the icienc: opening cficencs ; Gode-V is a co wy 4 enceding dhe Aone Arabeh, as it Code ta lew than thok fn code-T 22Dr.S.Bhargavi_Professor SWANNOENS ENCODIN g AL orETHM . following ane he shps dn the Shannon's Precedure fn grestng binany Codes - Step No 4 a non: increasing > List she fovrce Ayrbels Sq He tides 4 | C decreasing ) probabilities 2 Gren Se $s, s2,---, *44 4 ey : foe fee ee > Py STEP No.2 : cn ee > Compuke dhe ott a, 70 eee ee oo ee ee Cr ote Pat %3 : ‘ $ ! reat ares Step No-3 : = ey ng 5 Dukemine the denalleat foiger value 4 a aly 2 2 a a+Dr.S.Bhargavi_Professor step Not : jo the decimal number %; In binary hom oo Expane bayond Ai places. ope Ly places neglechng expantion step No-s > Remove ~the binany Pr PRoBLENS 1. Apply Shannon's encoding al gavitinn dolloning meager - s, Sa Sa os 0-3 Cue Find code effidency and sedundancy SovvtIon — Step No.l: Lisk the | Bouts symbol probali lho ecreasing Sy o-s a tal usDr.S.Bhargavi_Professor Step No-3 : - , oa Detinmine the ponelleskt inkiges value aq Be 4 the inegpoks by 26with cary | bith cone (Go ° = Co-F), = x, 20 and bye! ele eye Xo 2 (1, and 4.2% oe oy = Code Ss a The Code (200 ede and Hae oS lo bayie Source table Jn she mow be taritan as can 24Dr.S.Bhargavi_Professor Gay. Avenege length > @s5) WO +3) + lo-2) (3) 1-7 binit | rnenage trated = 3 Enho > Spy bgt Py : HOS) 2 a Ps ea See ese, ad a ore: Sl A eos bits | metoge - Apert Code eee 1,7 ME) Lc = [.Agss a oe - e1-38/. se Code redundancy Xe pi. Me 22Dr.S.Bhargavi_Professor Conshuct binany code fn the ltesing Jource vaing Shannon's binany encoding procedure - S2 4s. Sa 43 Sq Ss 5 pz § 0-4 0-25 oS oe 0708 f Sotuti0on = Step Newt: Fe, = eee bits | mrasege- Spnbed Code fhreeng qe ee als =e Saas) is. cody oe © 4oel- | a.ss-/. . i = Gi) The a my 4 Bs in the code ote P(e) = - 3 Lovtmber q ds in the code to «] i) fea : : Co toa) Plo) =i 7 cayconr eed Loa) + Gao) +07 nq plo) = 05263 — The pebebily q's in the code whe code youd oe et , Crd por = Number g hs in PS Lab 4 ; ‘= r + = [org + (o2) + Gear 9 6 ed pu) = 0-4137Dr.S.Bhargavi_Professor 3. A pounce produces too Symbol $, and $2 with probabilities ag and ‘/g mespeckvely . Device a coding Acheme using shannen- Fano encoding prcedune +o ak a Coding Yfideny g Sbleask sae Sovvution RO 2 Avaxoge Nengip ae 3, pits = ayer) O) iE 1 bins | menoge -Apnt Enhepy Wis) 2 ARs q 7 de: pee | yey ee yl Noe esse bel syebd ‘ = So cece 356. Code ficiency eo ¢ = S4 Buk ie effrccoy is lew than the to fideney 9 1s] a Now lek us Consider the 24 eakenaion 9 vhe ro Code «Dr.S.Bhargavi_Professor ol ool 000 = A, Averoge length ba = 3 yriki Te) 7 Ho +(2)eH + (2) + (a) Le \3sa33¢ bintls | mops -apabtl Eobopy 9 the 2) extended fone i W(s*) = 2 HOS) = 2(0-s435t) wis) = les bir | menope “nh Code tien The amt exhinded Ame 1 Ope oe) ee oe tL \asqaas a qtr than the sayin vale 151-Thes she 2" cchension woe is the Mpind coding scheme This value ( 40Dr.S.Bhargavi_Professor 4. A Source produces shoo Agrobel s ‘A’ and Sp’ with probabilities 9-05 and 0-45 nespectively » Gmaback a fultable biaany code Such thok the fficeoy 4 codingy Ts atleast 65-/. Sorveit)es B 0-45] 1 Tenath Gi in binile A 0-05 |0 Average length be te Sa a Coos) M1) \ toni ts |roenage tp Enbepy Us) = ah eileg 4 “ i + a0F beg os 2 0.2864 bits | manege- speed as = oas bg —— Coding ayes ae a oe c4 et - 29-64-/. es. This eficorg is lew than the rt enn q ah nlow lef us Canadien dhe aPd extension g He Gm? BourteDr.S.Bhargavi_Professor spot: pr babilly 41s) G@il+ 2 bo = (o-qo2s) W) + (v.o415) 4) * O° ae Co. 0025 = leas bint anemege- Hrrtet : Enhepy 4 the a eskinded some le geen Sate) Cs") (0.2% 64) =o anbA = o.g128 bits | renege = 7 . ood. yicery “| ts) cee eae : oo Vlas This Hh song 1s lew dhan the wn “ygceng q oof43 a] Dr.S.Bhargavi_Professor the 4a “7am Oo WO ° ke \ [seeeoe-9) fees a Now lef us consider the 3° exheneDr.S.Bhargavi_Professor Average lengt Li = 2, pi ly = (0-657075) 0) + (eos us )(S) (3) + (0-002875) IIS) + (0-000 125) LS) = 1.24475 ints | mnenage- arated Enbopg Ws) = 3 HUD = 8 (o-as6) = 0-859 bits [ete Code effaeng yO) 2 HES) = oBSID 2 Gb.) . sas ba 29415 The Heng is pe han the segpired value 4g ot Thus the ad extension code is the dagpined coding he SHANNON - FANo TeRNARY cope Dee ee eee ie oe Tre pradune is a4 pile = \. The syrobels oa. arnanged acending to nen ~increnting prbebittties - The fpebelr ane divided into ance poops 4° Sum gq pabsbiliier in exch pe apprieately yl AN the fymbole to dhe UF group ane cleigrated by a a eed ger by eee ee 34 poup oo AsDr.S.Bhargavi_Professor. 4° The group is ogame Aubdivided indo dhnee male Aub- groupe Such thot each sub-qro-p probabilehes ane opprtmate ty Pare + « cleiqn-dz. AN Apobele 4 est Sub- qtooP on el 9 a by ae ate at sub-gteg by aN on do the Bubdiided in teen sa epost - 6. The ard 4 ard gore te mBe sub-groups each and step rio-S H: This pres is Cemfinued Hl gutter Sub-divien PRoBLENS < d th (eee or ehannera Gene Feanany code fn diwing ensenble and Hrd code apiveny Also draw he caasrpording code - bree redundancy : Se fis, si Ss S4) Ss se Say oo P= fo.3 os, ote ole oot O08 5 wth x foyyesDr.S.Bhargavi_Professor SoLuTION Ls: a (ee) Gl “21 = (0-3)0) + G3)G) + (oe BD (0-06) (3) + (oo6) (3) + (04) ed 21-56 late binih: = yb L 256 + | menage ~ 34 7 t - 1A Pi leg + Enbopy Cs) ed = yen Sas leq ee: fog a3 leg 2 ° 4 + 0.06 leg . HG) =Dr.S.Bhargavi_Professor mbol #% Buk the enbory fo F-oy anits | menage 4y is Gia by 1 i) 6G 8 eee ea? 2 15452 teameng units | enenoye 2A2: log Be etymbl acs) ca ts quer by % Temony coding efidency 4 2 1ao0s/. 4) eee e998 = le « ao se Cope -TREE = aqONIT -3 FUNDAMENTAL LIMITS ON PERFORMANCE ae Source CoDING THeoren HUFFMAN CoDINg Discrete Menory Less CHANNELS MotvaL INFORMATION CHANNEL CAPASTIY aaa ro aa So aaaDr. S BHARGAVI, Professor INFORMATION THEORY & CODING (10EC55) E 4 en Source CoDING EOR Fiest THeoREN ) ation 44 dota gqueeted ies ( SHANNON'S % The preee of efficient nepresent th called Source encoding - bya Aiserete source dulien %& called The device that peynms the nepren A Sounce Encoder Discrete Hemayylens Sounce Laament agplemest a et ed ~ Linaay fam: le unignely The Source code the sriginal pouce Aegnence from the ence lea) biroy perfectly discrete ema founce Consid on in Fque TT Dept of ECE,SJCITINFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Profe 7 |, Professor ak Lek Su be the outpuk from dhe Source need + be encoded into a Shean 4 os and ts, denoted by bye. ee ee tha te 1 aynk ol we) with K difeneat aynbels 20), --- Su occurs with probability Pe KE), hel Si A Lek the binaay Code- nnd length te sgn by the encoder hove Neng Lie es) ea % Then dhe average Code-w1ed leag uc 4 the foun entode, can be Daitten of ce =! ele a ia numb 4 bh, Whee TL nepraents dhe avery the foure encoding Poor PA Source Ayre aed in ma Let Lint denote the min mum powible value 4 ve, be Thea "1 tie < unce encoder can Coding, yee “4 4 dhe 4o defined on CS ee encader Z dt te Zed be eficent tah a en t 1 : dieten 7 le abted Hence , the our — encoding ca a4 dollowt + Dept of ECE,SJCIT 412INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor * The Source -coding gaa giver fiscrete remaiylers ounce - ; ake alyatton a cele oe lets dhe ceereasing Heptman encoding reduchon proces pm ; a Lise the Sousice segnbols fa probabi lier cheew ‘dh; a et gatigkred a A-any coding hae oo 14 Cik aero binary ae - Tennany 4 Geen is the number 4 ounce hun bele i in and x in an inkegas 4 3 Tk He above condiven % OU isled ada \ 5 prbabilily - duumeng symbol iis of b &y Sa lat a Agyeobo\s as Sgrabel by addio 4 \s,' ‘ato a ingle we gk a aeduced Source % Dept of ECE,SJCITINFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor A. The last " symbols of Sounee Sate TPO Combined to fam another composite Aymbel bg adding their probabilities to gk = puwhe seduced Source “sp's The symbols gee ane 708 aananged in the adn 4 cleeveosing prbabilibes . a inal ate 5. Wis pween is nepeoked Se tee denel eer only “a Aaabele ane lye syne! oa werk mbel - 6. Aasign the codes to dhe last “9° backwerds +0 at dhe codes jn 3Y) qn code oid 1. The pocedine i covlinted fu Me “80 code wads alphabet 2 4 te s- to all the Source Ayrbels 4 discanded Coe a a Ss mnboo | Exanple 2 A Source produces Ss Ay With areapective pobabilities 4 ont, 0-3 0-4 0 008 So d. Gi Consbuck — Bulfman Vinany cote a (id Debeneine efffefeney oe aedundangy 4 Gi) Drow code-Wee 2 teed ten is wn tokeger Dept of ECE,SJCIT 115|, Professor S BHARGAVI, Dr. INFORMATION THEORY & CODING (10EC55) mevi-3d0D 116 Dept of ECE,SJCITINFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor GT Edger a Redundancy a ob: Average length 2 ern = (o-A)G) C03) (2) + Cola) &) + (oN) (4+ (08) 6) = : nits | oe bel 2 aos binits | menage AE" Ss i 1 Enhopy H(s) = BPI PA By jet Vlog ae \ 12 log — =? od leq te + 08 a Bo. 1 + 0.08 leg 13, See ae Jrenege ~spabel code ettuency A 2 HIs) = 2-041 yor: saa L 2.08 (too- Ae) 7- ee code Redundancy Ra. = (iii) Code - bee Dept of ECE,SJCITINFORMATION THEORY & CODING (10EC55) Dr, S BHARGAVI, Professor Pros.ens ith na XB MH, XS and XY od Sreapective prbabils ot, Om, a o ; oo. Conshuct a bineay code by aprlyig Pediee encoding procedure. detinmine the 1 51,80,88, $4, sc S05 Fo 64 ee le es apie x Cr-1) yee, vee 6 2 2rk2-') Gea 2 4, Ahich is an integer - Mence satishied 119 Dept of ECE,SJCITDr. S BHARGAVI, Professor INFORMATION THEORY & CODING (10EC55) gravy - 3960p 120 Dept of ECE,SJCITINFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor 6 i? Avanage lengh bs ori = (We) OG) + (ue) b+ (te) +O Ya)(s)+ (te) PI (ie) &) . : sails | epmbel azdse bnk]q Eninpy 4 C5) = 2 ni keg = ted i at > ieyegans (Qprae rt Gee Gl Ue) eg * (Yn) "8 - = a.at4 bite] synbd .23/f. ee So) ae G ae oo on \- Me (106 - ag-23) Dept of ECE,SJCIT 421INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor Note : — NAR LANCE aa % The Heasue of the Vantability tm avenge cedle- nerd lengths of * Bounce Code Lb oven the ensemble a Bowree Aymbols {% Known as NewWance + Uhere po py ane the Bouce probabilihes : cae tcl and Ly iz the Legh q the code-wad onigned to ounce Ayrobel Sy amd Lit the avenge length - The Vaniance o> is Smaller, when a combined Aybel ie moved as high a+ powitle , Companed FP hen Tk is moved as low as powtible - A,6 6,2 ard e vith, A source emits 5 symbole ee eee ee Sw esible G) shykng the cerbined syrobol A high a= re Gi) shiyheg the combined Agrbole a low os powible yp the coder. lan’ ance be Git) Fed coding etederey and + Dept of ECE,SJCIT 422123 pits) xq ebltl-o = (se Professor 2:0 we 0 12 te? py, oe = ba <0 + 7 *9 "2 e.y Vo sue oe quest (270 Rdaqua Dr. S BHARGAVI, pris Pome [ang ee (Ce) © Ca) Ch (2) (0) + Ce) 9) + 2 : Ws Z INFORMATION THEORY & CODING (10EC55) Dept of ECE,SJCITProfessor Dr, S BHARGAVI, INFORMATION THEORY & CODING (10EC55) He ees] at Ebitl-t Boe ee eel ea Seo oc ae he oPosreu | suid oe (9 Cro) bey 2po> ay 2pe> Ve BENS ee aesan Fyoqe ro1enos aa" is Based 1c)INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor Gi) Edpictency and Vanlance Eppicerey and Neniqnee ot w — Uy Effreeng {= Hes) 2 2.12193 2 96.457. Wed 7 — patio i) a Vontance = 2 stip (&-4) jz0 ae 2-2) yp (0-2) (2-27) = (0-4) C2 oo) Co-2) (e- 7 + Cos) (B22) Lot) (8-27) OAG = and Vveniance 4 Gi) er an Gi) Elecen = HGS) = 2-129 4 L Be ie! ra 2 api (&-4) cee) oe Ve 2 Cod) Cena te Con? Goat) eed Oe waa)’ p Cot) C422) Nosiana eo + Cot) CA oe = Dept of ECE,SJCIT 125INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor 4. Design a Hanony source Code J the source shown using Huffran's coding procedure . Ss 45.52, 53, 54,85 S23 ot 24 x2 fos2} ye rr ede) ge 34K (3-') G2a4 2% aes 6-3 heh i not an as ee oe inkge eee be need + oe Aurory Since ole grach on symbelt nade to mae “wan tokeger - Woe FO becomes an integer only 1 4 ~ oe 4 in the wt higher inten te 4-6 acd the next hig % q am Sd Seppence 2 1," One dummy Aymbo! de Pi . oe noe gollewert of ro ae fos Huyfron coding pre combining Last 3 Ayrnbol+ using the BG aiphabek x Dept of ECE,SJCIT aeProfessor 127 = BBV - 34eD er | 100 Dr. S BHARGAVI, sroswhs Braces INFORMATION THEORY & CODING (10EC55) Dept of ECE,SJCITINFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor 5S. Design qptartinnany Source code jx the Aovsce Zhou using Humans coding precdunc s-f{s, Se $3 S455 Se sif pe{4 3 3 2 4. 3 3 ae a ee ee ee a Ppp ee 2 es ee es x efor sy Aed find coding effieiensy Ye V4 KOr-1) oe eves ae feels) . = 2=3:) yt bt Bx Dp BAT + > AES ane dumeoy epee ene neanony {|__| Cove-Tasie For Qunrteznary “ove ee ok Dept of ECE,SJCIT 728INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor = The average length Fe tet FA aye Ayr er Bui 32 32 3r 2x2ar dvr + dBxrt 32 32 3> eee Br Az Br ee ass be ie cate aon ta] mmettoge” Apa a) + BU) \ a og 3 Ws) er ee ee 3a 4 32 q Se a crn 2.56 bite | Geb : ze Hats) = 2:54 a8 apo leg + = rege ia fate units L Coding ffiseny 42 Hyld) — L -26 /. 45-26 / Dept of ECE,SJCIT 429