AIMAKVA cuUEnierr sn - a & ATHARVA COLLEGE OF ENGINEERING, MUMBAI Name- Dev Nitin Netarky Poll no -14 Branch- CMP Bon Arsignme att a 4g deta ond enplarn “in dedu'I >|) Explain Par _cheracle wsties of bs — r | volume Cstructored) Covrze of dalaels - | artery Consructeved” [ Loy } I Data] 7 = [Lc Reateteme) (hate ma, “Ve fours, Veruuts. WW] Volame DIvolame we a hyge amornt ot Ty] Fo determine the Valve of dua vole -whetler 9 particular pele. Cam ealtually be is dependent upon tHe valu? TY Hence while deading with Ory dda 1+ ts recessery to Gorsies 2+ data plays @ Cructal oF dala pieys—a caverta an Oyatalalerdnus Chara Clerist re > volyme' By the year 2020 4he qlobas mobile tral was 49000 Brebyles of dea: ey: | velocity « H Wels erry velers to the high speed oF artumuletray of dade Tn B's Deva velury data Plows in Pom Surtes Whe Ea) ieee nes, nedwoy ky coci at mrdiray mobile phones Isle z )ghe potendrat of dota @ that hoy Dis ¢ 4 ines Wy Tes dete ’” a) and prwsed wo meet the demand” ab » dora rs gener fae arches perdiy are Ma ey There wre mE dnan B'S dillén Searcher yf de on Google * Vaxtet he Stracty I ech ao notare oF daa thab is Stace ene na En) unstractered date’ TH) Te otso vehers to helerogeneous Sources that By Variety ts basically the qorivet cf data Pom new Sources ore bot inside and oadsrde or an enter prise Ts can be Sprectared - Organtved data Unstractared - Unorganized date Semi strectared~ Semi-Oryani ca data *) Vevout By) Ty wefers to inansrstencies and uncertarnty ty dade, that 1s data whith is available can Sometimes get maseeve me and qtatity ard ecccuragy are drPrcalt #0 Contre)- ey. Data Mh bulk ald cree Coniston whereas lea omoint of deta com) Convey hall information Bb) Pysttm gutsy between Mame Woe and Data node - Name Vode Deda wode T) Trrrs eso lerown og master node DTI & alse Rnown as slave node of HDPS of ba Uprs. D) Handles He metadale oP atl phe Giles in HDPS and controls He dara node |B Storeand vetrreve brcks Accordiny tothe master 7's Taste Seth n + | | By Name node ha sob tracker whl} Keeps ack of Die agar’ dot e) | to Pada ndes | Duta node takes Client address mm Mame Vode-PAGE NO. ATHARVA EDUCATIONAL TRUST'S DATE ATHARVA COLLEGE OF ENGINEERING, MUMBAI Disa mbde other than Nanr Di] Dis a controller and manag et oP HOES Node fh HDPS that ty }- Controlled by Name Node> ar ©) [Way 1s HDPS more suited Por applitations havng large -datwets and not when Phere are smut files + Blaborte HDPs ( Hedoop Distvibuted Fle Sytem) & dargned By hordhng lage Ootesers effcrentty Bat is nor well sufted Per managing @large no of gmall Piles” = MI] Bree Sse EPlrctency: DI Mpes gteres deta ty dicks, wih q default Wouksize Wwor'eally Sed to fae 123 MB dv 256 MB. Back files, HOPS 1s broken Prto blocks of ths Ste and spend ccsoss dl nodes ty He chaoter For smal file, however, each Pile sicll occepte afwll dock - This ‘leads 40 thellvcron t usilzation of stove e Spece 2H] Nawenode hetada-. Overhead: T)| HOPS relves on the Namenode 40 manage reledeta about He blocks ahd their locators’ D|| Por smal file, the Nowenede hate manage a ds pre pyrtionelss layye Yet of entaes Leading 10 ex @esy've memory Corsumpho ant slower perfor mances 2] Repiscatton and Smatt Piles Twos repircctes exch block acroy multiple nae s: BH) [|For cmath Piles , yeh small file sill gets: wepli'eated, 2dd'ng toniPreant overhead and Woating beth Sto oes J netysre. betel” banduvidth Por repirreton of dota *a2 Sy ih dete |’ b) Whatare core Hadoop comporents ? Exgla'y Fi Je System CL MDPS) D Hadoop prsivbred Ble System (RDAs) Toe Fagaed jo scale 1 petedgtes of data ard rens on Commodiy hardware ~ 2) Yer Bnothe? Reyurte Nesattatos Cc ¥AnN) Tres yesporsbe tbr managing vesoures iin He caster and sche dung josus 7 user 3) Map Reduce Prosaa mmins Model Dees designed te process Tare volumes of data ty paralle | by devtdeng Pe Work Indo a set of Inde pare see Hadeoy Common: Fs provides Ae necessany Dave Piles and serdg i a Sa ipts required be Hadoop, equlved 40 Stard Rd a) Write * fhap reduce Pseudo code Ry eee Prosiems - Titastvate with an example showiny ol the steps Pseudo Code Class MAPPER method MAP C decd a doe J) Por all tem £6 doc d do Emit “Ctetm t, cout 1) cles REDUCER. method REDUCE C term t, sum <— 0 Por al count © © Lounts Cenes 3h Sam gum $C EMTT Cterm t, counk.om . counts [ ci, @,.-7)Type 5 vy hla P Hs i | Tp DvP png shhh hn glen Ehell- : E 2 D bar] Beer] py Heeb | 7 2 ? wy[ Baer] Bibel bert Dbar| 2 avd, i ; a Ce! Dek x [Bod r | Piyes] Toh jo Caly day} Piver: T_¢a Caer Rv eet teh | iN Defer | cxlr Beal 4 Dheek th |\/ Al \ al Dder| Ct | Rlvey Ctp Fret we drde the pape input Info Pree splis BD) Tis will Yisdaidute the work among Ul the Map odes WW) Then, we tokentze the words iy each oP the rappers and give a hard coded Value C1) +0 each of the dokens or words W) fhe Yetbomal: reason behind giveng he hardcoded Value egues go Ves th every word 1 In seself will occars once 4 DD Now a hot of Bey value pair will be created where te key rs nothing bet the Mdturdaal words ond Value yore TH) Tre mapping process rema'ns the same on all the vodes WT) plier He mapper ‘phere, & partition proces packer place where gorms and shufhling jappen so ‘fhatatl the teples wrth the same hey are Sent +0 the ener reduces ) Wy So afier the sorteg nee a XJ phase, each reducers bill have antque Mey anda hst Values cstrespading to thet Ver key- |B Bratt att the octput bey value pairs ave ther eke r watHen jy the oatpat Dile a3 b) Write q map reduce pseudo coe 40 Traltiply two "™MedrI'ces- App) mop reduce working to perform Borrowing micdai x mult’ hicadtd” ! 2 3 1 7S ¢ 2 > 2 4 e The Map Panctior ” Por each element my of mh do produce Chey, value) jrairs os CO, OMS) my) Br K = 2, bp yo the number of Columns of N° Por each element nyk of Wide i produce Chey pvalve) pairs as (aan >, CN, or PsKd, Pr te 1 2jd-° op" fo the number of vows of MPAGE NO. DATE —_—________ ATHARVA EDUCATIONAL TRUST'S ATHARVA COLLEGE OF ENGINEERING, MUMBAI The Redace Penciion Por cock bey CKO sort Valse begin with ™ by ort Values dehy with Ndy sim Lie Teacandl ne fo i multiply mig “99 wee Py Tyth value of Ccck list eg Sam up mh * nek’ jn bodys yetury Cie?! Ziomis ¥ oye $31 - A 3 i 2 I 4 Ss & | L 2 (—+—+ oo |x J es Matrrx Ary ( 34>) Rove C1) = 3 Columns GD 53 that’s Jr's Crt) flows Cp)> > Colum» Cup a) Mopper Pot ruta * Y Penne x Loy oar > A BCk v= Cork), CA, y B) Manner Rey matnx B Blew = Comer, C85, Bk) d Br alts Diy \) fore! kk s Mapper Lov Madre A Wee) Sal fe Cb), CA 4) CO, CA, 2,299 Coy CH, 3,9) Cor) - CA, 1d Cran), Ca, 2,52) Cozi, (a, ac3dCow «CA, 1,72) toa 5 — Cand, G Art 1.425 yo> CG,1>- CA, > 5d Mapper Qo mate x B- ved jen Peete CCl) IPG Opa seyI) ies eal Ctr) 10%, 2,22) Fee! Cewd, 68,2399 \E2 ye! wet (2,19, ¢8, 1,199 hel C @,), €8,2,2)) *e2t (CC 2,19, C8e 3,33) ve 3 kal C 3,0, ¢8,1,1) ka) C041, (8,2,299 ) ee) C32, €8,/3,39) Reducer Pormwex Reducer Ckrvd> Cie): Make sorted Atist & Bhat Crk) - Summation CAty ¥ BYR) Pr y C= Ribs CA, 19 042,29 643,35 Bist C8, 119 CB,2,2) €B,3,39 Now Aty © By k = Cix td 4 (rere + 03 dD = 1+4r9 = le —_.a') CA) Alisd Cmyi,4) LAM SICA ED Best Cig, te #2 (Br2, 2) C8, 3,3) Now Aiy * Bjk 2 CARL? + CEH SYD = &tlo te . = ot wun) C3,0y Aiat CR, ATS CA 280 CM, 2/99 Biest CB, U1) (8/2,290B, 3,2) Now Ay « PjePAGE NO. _______ ATHARVA EDUCATIONAL TRUST'S = PATE —_—__—____ ATHARVA COLLEGE OF ENGINEERING, MUMBAI > U7r) eC By2) + (9 x) ee OFF, = $0 — cid) Prom (0) Ur) Lid We conelde Coy, ( (210, 32) (Cap), $2) There Bre find matin ts (aera) [sae | Use J RF al] Bxplaly natured y'o'n and groping and aggre gatd> relation ad olgebrit Operator lotng rap Reduce Nature Join’ TH Merge two ‘tebles bared on Some commen tlunn DH The otpy will Mh wotaln vous Py whch the valve «> the Commer column mMmevich es * D Lt will goperate one rov fr every Jim die colvm aur ACTOSS Wo Jables* ex: Nome | Ay Joho | at Tom VW iva me. a nuh le an 7 Saar ales i Woe _[saah | 16| Name | B Ache | Monty 4, pn | False Smits Troe Tom Trae * D> Tnabwe es: iP there were multiple “Tom Values ty ase table then Rar rows woud have deen erected i» the output table TE prerenti'ng oll the Com Sthatha ns - 22 Gemupng © Aagrrey adion 7) Grorp vraws bused on some seto? cohemmns~ an) apply Some agg rey adil Csom, coon» Max, bpm d } a : M™E loamy oP the Smeu! gra. thatere Pyrmed > ° aia ey: Nam Winm'ng | Toho 200 i" | Winaty Yoo 190 Gm By Uveme) Bon 9 Smiby | $20 Ays Ch Cena) Jom Bvo vai ke | ae — Srish neal Soh» | Io mite Boo" Smith | 200 To oy 4oo tb List and erplain the wre business drivers Ledin J se NoS@L Ypove ment 12 Scatebiilly- BP Tra ditronel relat ral Iotabases often stragflt $0 seule i OSS Yhohy Servers hort conte! y oc i ' ; T) NoSrl databases are destyned to scale out easily , distodulrry mattinle woves and handling theveased loads daa across p 3 tbat cignt Preant charges 40 the eypplitcation qrchecre 'PAGE NO. ATHARVA EDUCATIONAL TRUST'S —DATE. © ATHARVA COLLEGE OF ENGINEERING, MUMBAI Ls || Flesi bitty” TF D || VoSap “Volabeses Support th var'ety of dada models, which allow Pox Perible schemas and easier hand! a of enstradured 2 emiStractured data’ TH thes Flegibiiny os cvucteu Py -appitcabions phat need Jo evslve qurckly dy hale diverse due Jypes 27) Brg Pata Handy, Bll We see oF Die dae and read-time analgtuc has dm'ven the need for datasases thet can elfictently manage large volumes oF daa: TD) wosal dakebare are wel-suted For big datq applications due to they abiliig 4a hardle lavge-scale dela Storepe and processing. al High Avetlabify and Fault Tolerance DIL mary We Say databases are bullt wth dict buted archrtectures that inherently prowde high cava'labrhiy ard Rut tolerance tt) || Dey often use replicator ard data partition! ng state gies Fo _ensyre that “dela _vemains accerrsle even inthe event oF hardwore Lar lures +2 rs the three ways Th yeoources Con be shaved detvee “Compuadd oe) Name the Systems: Name the architecture C2) In big data soletiors ond desord ) ia in Jett}: Shere) Memory System C UNDER PSD ) ®© ? ¥ To er i cedism | . Network ) le _ bad | J a =a) OMMen Jo shared Memory | Pera pei ale . lo] [ol Jo] D MAME CPU are attached toa Common global shared men ory Vea Tyter connected netwosk oy Command (ation medvork FY Shared memory arcritectore “owratty have large memory cache Ob ear, rec essor 2D Shared Drsk System COmmerE Macy: bE» | | im ( -@) @ ® OF dee communteadiny Networks — { Lo | [py Pe! TS) Madttple processor can accers all disk wa Peter common newwerk Buk every Processay has local memory TF) This Yn pres taut tolerance Zed Memory I> no bottle neck +PAGE NO, ATHARVA EDUCATIONAL TRUST'S DATE “i ATHARVA COLLEGE OF ENGINEERING, MUMBAI Shares Wothina Disle System Crores BD a (+ Trey commmtcativy: Ss Networle © wz — L 7 | e © & “| | | Im | Tm] Im_| DI] aD D Tl] Boch processor has by OWN Memary and Onk lows $0 high Scatabihiiy and partiel | Ty] A proceso? ab one node rmay Communitate with athey protessss | Gsing high Speed Commernicativa network - | i AS ml Descrbs He For vagy py which big data problems are beadled | by We SRL * } MIL Howize met Sreding * Tl esa L daub oes handle lane amas oP daba Sy Stalng horizontally mecning Peg add more Servers or nodes to Distribure Me dara across Multyle mache: FI Dis cs difPerent Rem Veraeal Stdrg where you add more pesowrces toa Single machine =) Horironiat gcedng thSuyes that as devta yolune sncreasey a the Syshem Cantbargle He Joud withoud bold lenecks-a) Flevi ble Schema’ ad Jababade, WICH YEpUITE S pred s\ TY Unlike radrtton reladon aii schema Por storms daly, Mo SRL Aatebaoes have & flew'ble schema % HF) Ths alias developers te sive ofherend dypeo oP desta, add re Pields) or make chars ¢o without needing +o ater evrst'ng stractares* . BD) Ti's especially weld Br handths unctractared and Seml- stractared Juda” a EPL cert Pate Stores ¢ Lov PPP. wodels- )5) NoSals dababares Suppast JAM. dela model, Such ag Mey vlc document, colars- family, aud graph models D) Ths Pew hts allows Hem to efPrctently store and manage varies types of data , depending on the we cae’ BW) Pow pnstance, qocament cktabares bbe Merge DB are well suited Ae OSo Nir ke Structa re y 4) Drstvibute pata Prog Tamrac ally * T) NoSal datedarts can Aignibule Jala acras different odes pregrammabically » alow'ny for better Jol beloncing asd desta x edundenc E> This dist bution 4's offen bused-on partitions stradeyres , Seeh ag a6 Yanye- bared parfitfomny encarg that date rs 2 PProtentty acres “nodes for perPoy mance ~ hashing s prea