24 /ortrer% — Archilectere and Tools~ ADs ‘Lineor Processing” Ihe problem in i there’ on core, the hele precessing will Stark olor from phe bagh ny. Bad in “Pasellel Trecessicg” the tesue nerds be he feed Sly in te relevent instruction. Th Bip Delo xe canned sede up Corrdico] sealing) after o lew) dee te limifedins. A @ selu? heriaondal seal Perkermed, # Assure Hew '!s a set of nunhers tn cech storuge node ond we need to fad Joe dep 10 of Me lisk Tn bere eech nede can process dala end thd Lhe Yep to numbers separdely each node. then ve cay trersfer bese bop ie values to 1 rode tres mode and aller thak we com sort phe munbers and tnd tha fra) dep 12 rumbers I's This can be called as ‘Mewly Bx Peel Mere ve bese fo hendle numbers 01 was dong by HaP Reduce, bed row Sark Vs sored a Mery bigdada avolyhes Sotasks ore OES A Open-torne meckly ave used Ge By-ulo Lecowe bhoe s emyh Stacked Scala, Cydhonti decmentatien to Gnd, ih bus become a stanchrd and oly thts Mente fo mecliy and bendle cede. Tibeduchen Jo Nadoop ~ + Based on Google tyelen Cohihepapa released in 2003) Ta Mep Reduce, Compulefien cil be por Bemed in jhe thee whens dak is stored, é 31/0) [o0a3 HDFS~ ond Shaky! weledada + Cordination payt, will -be performed by - jhe and dod name necos will Le stored i dade eda + Used fr Saleh recersig & Useally has 3. copies of data. , 4 Bit rb con happen due Fo lov dime wraye and effect. megnotie21 /oa/029 Design Pa berns~ Top deon summariaaticn~ moe, sfandard cevighes ® Sumwarizapen pottems could bo apple ro teu datas | get an joa about the davrced, 26 /oz /nor3 =I Filtenay Rabterns~ MX besed on tho scenarte use Mappers, redycers S, Filkins padlerns is considered. he Cbpeh te not charge al Dilincd> For geting onigue volees Bhan Kleving > Ups multiple harh kncters to ove the output, Hore elhicient endl guicher lable with bluor &ilfering 1's you teed te cache tCstore in bh memory J17/03/2023 Hbese FA debsbase aveilable in Uodoop E Base js installed eth Hadloop (no need yo Separately inste Ib) ot Pole is acteelly stored fy HDFS. PE BBase supperde consistency chick allen it fo be veel cot di ferent applica dons. Clon Ferily needs be deloed al te Ieynmg . eraixe phe database needs Jo be resderte! when a column Lmily. fs acted in phe middle. Glan con't be tn cudsde of a column Lamily.Key Teves + Rees — Lnperdent in lod beboncing Clas Celene ~ Frost a colemn fomily P lodce a we mee inte colar 12 Clana Fomlyn EE fhe cerfont is tre lecge, dae con be opi 2 Info sorrel pods, Date con be spl int colimn famibes end stre & ditennd Flex. This is why cone Goriles need be detoed at the beginng. af Hal le..verstens- 0b timestamps coe ld te rethiowd im BBase, Lei be dene 1m tho lab sessing). [Base feedures= epiyh ovaileilyy — Hasler Slaw archifechire ensures. dhiLit: eve node tile anotter wil) Soke cvegh Trrovghtb—> bnewrd of defe)tresacdins flat tan te procesed. within a second. Regica. Serve . se | Dole ail be obbeied 03.0 ref edimus Som ther lege dale | cu and steeds tm one angen. shin Sed shay endccend: bey gor the Myien will be de Ged ond 2 stored 5 mete dele cP Tables ares horisontely gorkfoned. saab region seer con howe multzle rejens. | Me dul hee. gacdilee , reyen ab —pededlar Bae , dew "ll | so iche enlye te vegion corser =o balla ids Region. croned Lame | a pin Sersecy wat des come tine. :: pondlend unids: ive = | *Gannok rodiby the conten} Ina Sitg le raglan ty 2 | profle of fhe sone Hme. é + Regions. ove eufmeicaly..asrymed te. rgten..servers Ly Fhe tladerp. Gomewerk unbys ids morally castortned. Mes per + Shoul) howe eink arith dhe zookacper Sener + Mayes scone chagger > creelen ce lation Sa blery ally om removing columns,-charplag properties. Henages clusler Sfeles-> Spates off Hee region Sercers,, confine tons of te MMarter as/oalxo23 tet A lerforngreal-tne ond. batch. processing = F In Speak, Sndl_yorkins of dale are hendled by telirg indo RAM Crealnsime processing J - ee 1 i Faster hen Map~ Reducer. Hlerest= Near rol dime gp hen a Fine? x used hoy mul ple. processor Spark keer the fine? ia Mo RAM moling th Lester Bis chesn')hagen i, Map Redcar. Memory sro saved) # Lerglls. of Codes ore minimum Compared. -svith. ep Redce | SE | Sack, RAM is_used sjerdoom seperation: oh 6 24 - Tne lengthy — proceorg, Crh ing addin, mel dphen® wie ote J, inane (ae lion ta tara perloemed. va eottul is vehurmed, They is. celled ag 1 [aay evaluaticn), Srmcessng cull mof_bappen,contl “ples doce Cllis ts im flo nd), wrsceedlK Optwizatin shukyies ere edlicient jm Spanb, Aeborn® Floce shere the acta! Empuda ton te door. 2 Toon or-pytlen ‘can be eased’ in Spark. $90 cambe used tefiewe dado. * ce seyttiscs, used tn Spark H Stavk cm be nun indeperclanlly or it con be rum om @ deep bezel Ch) wil be. usa! inthe. lab. sessions de run Spon kd 3H Dingo ile if Ue driver redo Geils, system =i] Lil do incon properly. 7 ae Shee By Toshedeas /dPT. ley fer managers ¥ Sal CG PB RDO is a bro-lenel dee shuchee Im sph Dathlomer aren high level dade rtuc tare. 3 Diver Prgran is like Nhe main program. of Stevk - ashere Lyew delino jnpuds snd olben Hogs. a. Dy Spork cache 15 craked since in-memory operations lave -pecbormed. “ROD ~= Being immetedle , POD iD case a bodllomech when abclace heppens thin dhe systemDob E Foner « Osho in PDB, Jeno ban. Te Se poe opheieadios oe HT, Spork, REP is sphid inte seal Pees 2 Se be ee ooTe 1, Lecture ob = asstgnment- fe Se Js Map Reduce A ali TS me SS spark 1 e proposal # ant find dhe dotaset Execuhon Sequence r tcedion master 2 more close to pp master) client ‘ommguntcale one program —5 needs | App mastery a zsomehmes 1 AM can CXE Cute mop, than | program ‘ Centoiney con have mulkiple _Centoiners —> set of resources hr memory network | CPU ete Pseudo node cluster —, eae node cluster. Heart beat Messages used 45 indicate ot nak, whether a hede ts achve 4, "7 ssoBiteL HFC — Horase of files MR — computation /prowssing MapReduce = Mop Reduce data locality G ad the place where. we hove date proassing is dene Ciniial pressing) yw otherwise unneccessary burden cn the hetwork - do ene node — when we have a large amount of data transferring all ef them toa centre \ place is costly ond will vare network capocities. TMerefcre initial prowssing vill be done at the node level. a usage ccenarioS Filtering con be done ot Map level ae ly Mapper is used mainly, Reducer ts used only te sort and unite the ouput # like Use Yap per only (mounly) Mapper used to ges the input and mainly use due reducer. * mainly use reducer only input > mapper Ccannol vrite evtput ) output > reducer Ccannot get inp) oe even WH we need only one mainly we need the other just to get the input er write ake ) Aaareaaten [summarizcis a # dota will be transferred4 ‘ 7/ “Tt SUMOBITEL » 1 million documents apes ol 4 4 5 Himes in each do sth a word “apple 5 Mes _ ID EALN _COcument : se 7 : total = 9 million mes Kes value oa a ee rere) 7 Res fa ones oe le 3> whiter Aba ID —> —> Peer : aa SSID CAL POR, aapple,|> mange, ile oes __ on the _reducer__side- 1 > ~ . 0 rere a ae = Huis is okay iw mapper side since be io ae aise are mulkile raappers Thardling a en “apple, oe ee he ___ 00 ©00 or less key value pairg, x we customize only mapper and reducer co _=but_ih vedueer side His could be ek Hoe aeRile ade fc = __problemane. ~ eT Tae rt ee gS Oy oighal iva) oolaotina ge Ziggy used _to_group hese key. parr mite =< foe Se “hs — Hall ese 5 million key value pairs | im put key long = big Mee (een er hse ; a taken os likes value : oY 5 | mis value — text ; eS — Sith the _gome key and. value, | __output key Gey Al os - han an ar + { scarier a integer: oP bith all the values in the S million | Paes eae cg Merah’ 24 aa 5 Big Ga ally mapper __5 oudput i £24 Pols, | map _ inpu i a en ~-—_machine 5. ig. mappers ne a cph feskeua— ote —_———-——~ | = Val ue -text —_—— 6p : tele ctel ae BA = fe 2200/9" 25.000 iterations me WES Sorte Wo WWW. sitmobitel Ikees local dato process. enhify top lo values In each node 5 ma Ppers 3 to values ee iy inp wt to reducer: 4. “5 7AOBITEL Reducer base class = ere —— filtering rs without reducergs: emall files as output ; (eg 5 mappes—> 5 files) fo write the Select — wh can Wave just. MaPPe put there will be as the no. of maip pers better vO hove o Ye output with oa single. File nput to_ dhe _combiner- ducer mapper outpu4 could be, available in the mapper ee omnount of data thot will be ducer, # woduce the network combiner iS it yeduce she gem to the ve _Areffic. wenn sitmobited A5 7 SLi Lecture 05 “ity Partthoner ~ an some coses we wight have fo uy. multiple yeducers to make tthe tor} efficent. etaen we hove to decide whi ch day be gent to which reoluer. tet | - partton the mapper outputs, ~ | mapper-rreducer->poritoner ? 3 reducers —9 3 part Honers no, of partikeners = No, of reducers, Crample + : Analysis should be done at a levels, age, gencler j at | reducer —5 caleulahions relevant for | age aeup age —2. 9 tahegories gerd a cabego ries Fer processing at he twa Value 1g cance, “ 'S Used for processing , pper side, tre ‘key | 4, “/suonireL 0} Par bvoning dene based on age: 3 reducers Mopper ~ PPine by Une read the input use spl funchon and identify gender. ow ut —9 L aender P a 4 rd (line received os value! ye Co value te to the mapper ) Pardoner mapper output 29 porhtoner input Check the age amd see fo which reducer Jne data will be sent. Mh Doesnt do ony amendments to the key value ducer will receive the. acta relevant pair. Each ‘re. for oO porkeular age eroup only. Reducer + gg : receives same extpo= input as the partittoner. finds the moasimum. salary for the | 2g ere: (3) Porditioning done based on gender 9 reducers. Goch reducer will find™ the tghest salary for one gender based on {ose gre eps- (* opproach is Meal because les no.of rowssina. Pr the reducers 6 Lag her leod for the reducers. Www sitmobitel Ik_ Par Ationer ath opraired ovly if we rate _ more, than. wre ducer, Combiner VV aa 2 combining part [fame as aa —— Fat , reducer) doesn’t do ony. modifications - _ wo the jnped_received. = - poss the same key-value pore 2 received out it vill decide : jhe _veducer_to which the ummarirahon, rb is 7 : ' | ae perations like mean (fveroge) eco as f 7 oto should be sent. . Povktioner oukput will ge through shuffling and sorting and then the reducer will do ~ the remaining procssing part. ons a ~~ x = = < S Ss ner for o operat Answer | by a stunt 4 . a ppt = _inpatsaplit dato. a apollo | output key - value Choth number | ob containing ___ single value, ‘gn pe ‘% snot “use ce i they) are | not tomviver > cl \shwg wesl\ For pli fe reducer 9 caleulahna wean for de 3 spl but pus Bren a coun Aer See ; = meon ae “ne sews stabi WOLUL- = no. oF values used do caleulote. men (Count?) mM Later in dec ead we cor r L jontity Ma per doosn?t edo: ain, aol ‘ned a » Any (pro ces sin ; = Sa Aga Teg aton if summan zation Whe reducing activties oie _—- e€ nor performed. Gust doing basic shuffling aide i 2 a whan we. lave. eile le med wig ‘ Aeoall Us reducers can be identity reduces, “aR tee dask uses. only. Mappers ace ee Sel k even_ wittrouk an identity reducer) eve AGE ‘ Yr y me Sokal will be different Sile—> a lo z e ya af Files. arin cele eases “gxercise_ ‘compute mean for 3 input spits. Wodam “gend seme and count for each mapper [i ____(s0 we can reduce the netuork araffic) — | tif the amount of dota ig less, we can send - whole dataset else we can “gr Altering data 5 moppers aan pavallly Ee ee Caen ws data, _ rapper-only tasks ean, -te_oet dire output In output files. ae reducer > \ output file, Ae NS ~~ have iderbty reducers required re.of “ww sitmobitelik * — LiaS Se ey — s F eS oF) ol , 1 ea(gapthe eplits) ‘ splits SLiviogiy fg mappert depend on no-ef Oy P Bea en 5 input apis = me value : the Map—Reduce Version | ‘= snput > amytiing for key, et : Hing 1s moneged by the Tok Mopper oni <3 key Gr gender ~ E | ale aie: works on individual as altribute nee | Spon sill Rh ede Velue — numerical ES responsible for menitorying anc out pat Volu —— Vesion 2 ¢ when using —MopReduce. pregrans oi mr >. Resourre er — overall management ofr» Moe a woy to minimize the amount © a Ls scheduler + app master = we . ansferred in the network . P (6 beng te ‘ h 4d be Ly residing in -slave sid ~ 9s much of possibley processing shou je . Design Podterns, Summarization Patterns used in, 6G: calculating statistics —s shondard dericeio, vartan ce > Mean _ ceunting SS eto Ny bettom n * Based on the iyvement we can use _combiners and reduce the qmount ef data that is being transferred in the network. » D) Numerical summarization » * possible when we have numerical data, if not even if the data is categorical we GN use this for counting. 4 cunt for female/male. Using, coroners we con make the process move effective for operodions like adding - Jone wilin the mapper ‘ - ain reduce network. troffic- Le eae distribute data te different y Ef there is more data fora key in parkeular, : lot of Acta will be acing +O sne reducer and the cther reducer will have less data to prowss- used for load balancing. : Reducer Each reducer writes one cutput file. | | 2) Inverted Index Sumimovtgation ss | -Used to quickly search for specifte content . | - Th gives keywords along with the locations of { that keyword, so we can easily trace back. | | *) Counters Not like solving a problem, but we will Use counters as a diagnosis tool to see whether the activity has performed successfully: (Compare counter values with real values ) “aw sttmobitel Ik28/2/2073 7, th 8M Lecture 06 A atte! Stern eee os output and tip PF os done 40 the data. te fop_ ten tee ah fob... are bottom ten > ‘xlfclearn_ Pa ees “mapper—> ony. Linge date types. Aor bey ond val er _mapper’ s ‘sai “rey “volue, chet Ayes ghould be sameas reducers “input key, walue daton AIRS be, ds a a * “Bloom Filtering > a) head ~ use multiple. le hash fun funckens_ and ae ~ F More efficient and ar tg oat issues have to. ‘keep i in 1 Ahe _memory_ ae do | de PICESSing Bunny ; mates — directly: Atscodtid Cle ative) ae onalysis (False positive) # Top lo + f. Mazel mappers = n acmEN omnouny of deda sem to reducers = nx lo 4. “seraoarre. should have only one hoe Th _the_ ro- 10.08 records ir in ies n ck split i is very much closer to the outputne: of records, hen _no point in sing. tt. locally becouse olmest yre_same Yecords will be anyway sere to dhe reducer. * prstinek - d ‘e __outpet Bile Site 15 inverse _ 5 ai ho- 2a adicere: eg a reducers —> _Odput file size (0. MB 7 & reducers —> cvtput file size 5 M8- gs te ty neem soepatoral) —=— locally _ distinct volves vill be ease wut _raultiple map rappers st con have = ahe same distinct value in that case they will be sent to the same Yeducer. wires Vimobted tk07 [03(7°°" 7 \ SMO o7 Leche Th ion ni : fe, etd te be denrmaied yy then we will have dae | but that fs okay cinta same type of Auta wa together. Based on the problem and Scenario the tree Structure (hierarchy) _ can chonge. | “Example r StackOver Flov post : qgueshon comments ! _ length answers might { small _queskons have smal) answers, Micady > pasha ; = Comments In the diagram , Ge Hees and Serking for datose) can be done using the. pest ID, ‘ 1, LIMoB ITEL put Mapper © data of different formats from various Ly woverts into a commen shruchure resources. veclucer > converts into tree shucture No need to have a combiner in this case. performance _ Analysis ~ y of data is skewed , we can use a custom partitioner ond spl the data between reducers . post | ane thas many--Comment s post > z assigning 2 pests t 2 reducers — more work for | reducer: use: a parkdioner and divide the dota of pest | between 2 reducers and use another reducer for post W2. x Umited a0 memory capacity avorlable - Partiti ening To decide on the p the ranges of data : ee sample data an fem each elata nede and then find mn and marx - arhhons 4 we need to tdertify Www sitinobitel Ik.4 ‘1 > Inside the mapper code ig $0 if we want, we oo snning j do a mapper-only ok. and wefinal_output. porkKoner —> 7 is compulsory to so —_ gutput_do reducers fe get Pe Final output. a Heele = SS — es — not the same chuffling if we learnt earlier. ehuffling why yandomi2e ? s SLIMOBITEL to anomalize data—s so that 34 parties eink bade _- Avack which data is taken | ee _ from where. - ; because there are some personal data __ 8g & From 6B —2 address | contact — er out ete i ie ES eopite Ale ae pee “ey EO value > whole record read —> fields - details reducer + Dak based _en_ key —> sorted _ 2 ste ni a ein Pabkehten sa ot. tk eames ~ Common field vill be the key of mapper __oukput ree e4 + Joining. “customer and order defoils b& key could be customer id. Ww sitmobitel Ik“ 7, St MOB NoSQL “i ——— Shared “Nothing Architecture © / - Shoring is done through the _nehsorp - Memory and disk space + ave inde ‘ ey Vertical scaling - —Aftera certain point, we cannet add more vertically. <3 a ; = Then we will have to add more yn (Horizontal scaling) odes Semi-structured data. —> eg XML, Column oriented + when adding new column, +f we used rdbms , it becomes & prololem os in what te include for thot column forthe existing dota (whether null default ey But for ‘lnm eee we can ‘ae ly adda data record Cf null ad keep blank and odd commas) _ Docum ent Store - “Can . ri = at not indi cote connections and relations. " q]03/2023 ty SLMOBITEI. Lecture 98 pase — Manly column- oriented but has some fecttures of key-value store. -first the column families should be defings and they should be fined. TH is a bit problemodic. VE column families are to be modified tater. However, columns can be changed easily oS wecess requived. - Columns, must have a column family. cells ‘. : can have versions ond time stamps con be used to identify the versions. Column. qualifier = cae ‘ 3 Need not be. consistent between rows - eg \ yew > ranean another _5 name, resi dence, ian telephone Column Family - When content is too large, we on split and store data in different nodes, based on the column families. ear 3 different files: That is vihy WwW is said that CFs Www Slimobitel Ik under 5 “personal Column different column families stored in should be defined ot the beginning:77 SUMOBne + when reading, more Than! version can |e” retrieved by Features —_— Follows master- slave. architecture. a b tive h es amount of dato that a Heh ugnp ea over a unit time menial be precess er 10,000 transactions /second Architecture _Arenniecre. Resion= eee es ner __ Deta_split into _muldiple regiens, based _ on Yows. = i a L tar: eh sO na ona key lo : a NTS ae hea + Hettonlal getBtionny = Rajon sewer can handle. mauliple regent — But. one region can be handled only by Yegion Server, Ben ri aad Consistency —> a people vill not be_allowed’® mee: The content of one. sregion. ot tne - ~ Same time (uses -lecking mechanisms eae specifying. the required version. 4 "YsuMoBITEL qosks of HMaster ore Load : balancing '- 2 ; Can monitor the lead managed by. region servers and if +o much, the regrons vill be ve-assigned accordingly. Cluster State management — pone wih the Zookeeper... Look. er Distributed synchronization a censure a consistent content within the nodes . Read /Write — -Cliert first reads the Zookeeper and gets be lpcations of ion servers: ; ; —Then those locations are cached and therefore _everytime the client needs to read /write from/to _region servers, 7o: - ___ Centactin 9, Zookeeper ¥s not required. — - If data lecations are updated’, chiemt when a client dry to access data based on cache ~dekoils it will be oa miss. - = Then the chent will _eaain go to te the meta _ talote stored in _qookeeper and get the new location dedorls sa -then using that detorls, by visiting the region server. Vw sltmobitel Ik data will be accessed47 SUMogirg, ion server hotspotting ~ Al iain se When en writes hitting One region er all reads are for one Pegion -qae can create a bottleneck. I - This happens due to uneven load Aistibus, -This increases. Wisk of failure berauce ve that one peode foils, whole PrPessing can be distugheck disturbed (tn ‘the clustey) Solutions = = i) Use random keys. cate — compromise the speed *f data acces, "dee is sorted based on these randon keys and ‘therefore tae most recertly added data might net be tae in the last rows. Be aged 4 41 SL™OBMm EL