DATA228 Lecture Notes Week 5
DATA228 Lecture Notes Week 5
Sangjin Lee
Hadoop: YARN, Hadoop’s
distributed compute
A distributed compute system is system which solves computing problem utilizing set of
multiple devices over network.
a
a
a
a
Distributed compute
• Recover from ll m nners of f ilures: node f ilures, t sk f ilures, network f ilures, etc.
a
a
a
a
a
a
a
a
a
a
a
a
a
ff
a
a
a
a
a
a
a
a
a
a
a
a
a
Elements of distributed compute
More advanced elements
• (St teless- pp-speci ic) Provide support for tr ic ingress for pplic tions
a
a
a
a
a
f
a
a
f
a
a
a
a
a
a
a
a
a
ff
a
a
a
a
a
a
Elements of distributed compute
Stateless and stateful
St teless St teful
Coordin tion No coordin tion required Coordin tion required mong worklo ds
Dur tion Tends to run long (long-running) Tends to complete its job nd shut down
• Not quite but pointing the w y: servlet cont iners (Tomc t, Jetty), pplic tion servers
• Mesos
• Kubernetes
• Supports more complex pplic tion types (st teful, GPU, etc.)
• Sp wned n ecosystem of supporting technologies: Helm, cont inerd, Istio, Envoy, etc. (in CNCF)
• YARN
• D t - w re scheduling
a
a
a
a
a
a
a
a
a
a
a
a
a
a
Distributed compute in Hadoop
Characteristics
• Accomplish d t processing over l rge d t (TBs or PBs) within short mount of time
• H ndle d t -w re scheduling
• Resource requests
• Amount of compute resources being requested: virtu l cores (vCPUs), memory (MBs), GPUs, etc.
• ResourceM n ger tries to schedule b sed on the loc lity constr ints
a
a
a
a
a
a
a
f
a
a
a
f
a
a
a
ff
a
a
a
YARN
ResourceManager
• Applic tion
• Cont iners
• Speci l cont iner th t m n ges other cont iners for the pplic tion
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
YARN
YARN application
• Scheduler: ResourceM n ger component/ lgorithm th t lloc tes cont iners b sed on
cert in policy
• FIFO scheduler
• Simplest scheduler
• Applic tions run in the order of submission: the next pplic tion gets scheduled fter the
previous pplic tions h ve been completed
• C p city is p rtitioned with multiple dedic ted queues (e.g. te ms, groups of pps, etc.)
• F ir scheduler
• Dyn mic lly b l nces resources between ll running pps so th t ll pps get equ l sh re
of resources over time: “f ir sh re”
• Tries to b l nce between h ving ll pps m ke good progress nd letting l rge pps inish in
timely m nner
• Queues re on p per unnecess ry, but they c n be used to incre se predict bility
• The j r for the sleep job (hadoop mapreduce client jobclient 3.4.0 tests.jar) is v il ble on C nv s
• m: number of m ppers
• r: number of reducers
fi
YARN
Demo: sleep job: rst app (mappers running)
fi
YARN
Demo: sleep job: second app submitted
YARN
Demo: sleep job: second app running mappers
YARN
Demo: sleep job: second app completes running