Setting up Hadoop YARN Clustering
14 likes2,030 views
The document outlines the steps to set up a Hadoop cluster and run a MapReduce job across the cluster. It describes cloning Hadoop from the master node to two slave nodes, configuring settings like the hosts file and SSH keys for access. The document then details formatting the HDFS, starting services on all nodes, importing data and running a sample MapReduce word count job on the cluster. Finally, it discusses stopping the Hadoop services on all nodes to shut down the cluster.
Setting up Hadoop YARN Clustering
- 1. Danairat T., 2013, [email protected] Data Hadoop – Hands On Workshop Setting up Hadoop Clustering Hands-On Workshop Danairat T. Line ID: Danairat FB: Danairat Thanabodithammachari +668-1559-1446, [email protected], Certified Java Programmer
- 2. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop Big Data Introduction Volume Variety Velocity DB Table Delimited Text XML, HTML Free Form Text Image, Music, VDO, Binary Batch Near real time Real time GB TB PB XB ZB
- 3. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop Big Data Architecture Big Data InfrastructureBig Data Infrastructure BI/Report Next Best Action Distributed Data Processing Integration and Metadata Framework Distributed Data Store and DWH Monitoring and Management Framework Security Framework Predictive Analytics Descriptive Analytics Prescriptive Analytics Big Data Platform Big Data Applications Hardware, Storage, Network Fraud Analysis Cyber Security Talent Search
- 4. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop Hadoop Timeline
- 5. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop Apache Hadoop Core Technology j2eedev.org/ecosystem-hadoop
- 6. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop Apache Hadoop Ecosystem j2eedev.org/ecosystem-hadoop
- 7. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop Big Data Platform & Big Data Analytics Hadoop Technology
- 8. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop Block Size = 64MB Replication Factor = 3 HDFS: Hadoop Distributed File System Cost/GB is a few ¢/month vs $/month apache.org/hadoop/
- 9. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop YARN: Yet Another Resource Negotiator Hadoop.apache.org MRV2 maintains API compatibility with previous stable release (hadoop-1.x). This means that all Map-Reduce jobs should still run unchanged on top of MRv2 with just a recompile.
- 10. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop Hadoop 1.0 vs Hadoop 2.0 Hortonwork.com
- 11. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop Hadoop 1.0 vs Hadoop 2.0 Hortonwork.com
- 12. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop Hadoop 2 Hortonworks.com
- 13. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop Hadoop Symbols and Reasons Behind 1 3
- 14. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop Clone hadoop master to slave1 and slave2 master slave1 slave2
- 15. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At master node: Edit host file
- 16. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At master node : Copy key file to slave1 and slave2 scp /home/ubuntu/.ssh/id_dsa.pub ip-172-31-1-8:/home/ubuntu/.ssh/master.pub scp /home/ubuntu/.ssh/id_dsa.pub 172.31.15.16:/home/ubuntu/.ssh/master.pub
- 17. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop After this slide, we will use 3 cascaded windows to represent master node, slave1 node and slave2 node master node slave1 node slave2 node
- 18. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At slave1 and slave2: cat /home/ubuntu/.ssh/master.pub >> /home/ubuntu/.ssh/authorized_keys
- 19. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At master: Test ssh to slave1 and slave 2 $ ssh ip-172-31-1-8 $ exit $ ssh ip-172-31-15-16 $ exit
- 20. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At master: add slave1 and slave2 to Hadoop slave file
- 21. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At master: add slave1 and slave2 to Hadoop slave file
- 22. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At master: edit hdfs-site.xml
- 23. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At master: edit hdfs-site.xml for 2 replication servers
- 24. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At all nodes: remove directories of namenode and datanode
- 25. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At master: format namenode
- 26. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At master: format namenode
- 27. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At master: Execute start-dfs.sh
- 28. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At slave1: Check jps result, you will see DataNode has been started
- 29. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At slave2: Check jps result, you will see DataNode has been started
- 30. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At master: Execute start-yarn.sh
- 31. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At slave1: Check jps result, you will see NodeManager has been started
- 32. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At slave2: Check jps result, you will see NodeManager has been started
- 33. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop Importing data into HDFS Cluster
- 34. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At master: import data to hdfs
- 35. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At slave1: review imported result data from hdfs
- 36. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At slave2: review imported result data from hdfs
- 37. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop Running MapReduce in Cluster Mode
- 38. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At master: execute YARN mapreduce program
- 39. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At slave1, slave2: you will see Application Master and Yarn Child Container
- 40. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At master: review output file from hdfs
- 41. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At master: review output file from hdfs
- 42. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At slave1, slave2: review output file from hdfs by using command:- hdfs dfs -cat /outputs/wordcount_output_dir01/part-r-00000
- 43. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At master: review output result data from web console
- 44. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At master: review output result data from web console
- 45. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At master: review output result data from web console
- 46. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At master: review output result data from web console
- 47. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop Stopping Hadoop Cluster
- 48. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At master: execute stop-yarn.sh
- 49. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At slave1: use jps to review NodeManager has been stopped
- 50. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At slave2: use jps to review NodeManager has been stopped
- 51. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At master: execute stop-dfs.sh
- 52. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At slave1: use jps to review DataNode has been stopped
- 53. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop At slave2: use jps to review DataNode has been stopped
- 54. Danairat T., [email protected]:Big Data Hadoop – Hands On Workshop Thank you very much Danairat T. Line ID: Danairat FB: Danairat Thanabodithammachari +668-1559-1446, [email protected], Certified Java Programmer