Scribd
Upload
106 views10 pages

Expt 1 - Hadoop Installation

The document provides step-by-step instructions for installing Hadoop in pseudo-distributed mode on a single node. It describes downloading and configuring Java, SSH, Hadoop, and editing configuration files for the core, HDFS, mapred, and yarn sites. Starting services uses the start-dfs, start-yarn or deprecated start-all scripts, and the jps command checks component status. The HDFS web UI is accessed at https://localhost:50070/.

Uploaded by

Nimesh Naik
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
106 views10 pages

Expt 1 - Hadoop Installation

The document provides step-by-step instructions for installing Hadoop in pseudo-distributed mode on a single node. It describes downloading and configuring Java, SSH, Hadoop, and editing configuration files for the core, HDFS, mapred, and yarn sites. Starting services uses the start-dfs, start-yarn or deprecated start-all scripts, and the jps command checks component status. The HDFS web UI is accessed at https://localhost:50070/.

Uploaded by

Nimesh Naik
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 10

Faculty: Ms.

Sana Shaikh Hadoop Installation Subject: Big Data Analytics


2020-2021
Hadoop Modes

1. Standalone(Local) Mode
2. Pseudo-Distributed Mode(Single node)
3. Fully Distributed Mode

1. sudo apt-get update

2. sudo apt-get install default-jdk

3. java -version

4. sudo apt-get install ssh


Faculty: Ms. Sana Shaikh Hadoop Installation Subject: Big Data Analytics
2020-2021

Press ‘Y’ and continue

5. ssh localhost

Press ‘Yes’ and continue and enter password

6.

Note-Before going further we need to exit ssh just type “exit” in same terminal. so we need to set
our ssh for password less communication. To do that execute following command in terminal.

$ ssh-keygen -t rsa -P ''

press ‘Enter’
Faculty: Ms. Sana Shaikh Hadoop Installation Subject: Big Data Analytics
2020-2021

7. cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys


To check we have bypass the password we need to again execute

8. $ ssh localhost

9. $ sudo mv Desktop/hadoop-2.7.7 /usr/local/hadoop


Faculty: Ms. Sana Shaikh Hadoop Installation Subject: Big Data Analytics
2020-2021

10. $ nano ~/.bashrc

type:

#Hadoop variables
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64
export HADOOP_INSTALL=/usr/local/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
#end of Hadoop variable declaration

ct+x... + y ... + enter

11. $ source ~/.bashrc

12. hadoop
version
Faculty: Ms. Sana Shaikh Hadoop Installation Subject: Big Data Analytics
2020-2021

13. OPEN (/usr/local/hadoop/etc/hadoop/hadoop-env.sh)

type and save and exit : export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64

14. To configure hadoop in Pseudo Distributed mode we need to edit following files
1)core-site.xml
2)hdfs-site.xml
3)mapred-site.xml
4)yarn-site.xml

14-1 OPEN (/usr/local/hadoop/etc/hadoop/core-site.xml)


type and save
Faculty: Ms. Sana Shaikh Hadoop Installation Subject: Big Data Analytics
2020-2021
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/dbit/tmp</value>
</property>
</configuration>

14- 2

OPEN (/usr/local/hadoop/etc/hadoop/hdfs-site.xml)
type and save

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/home/dbit/tmp/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/home/dbit/tmp/datanode</value>
</property>
Faculty: Ms. Sana Shaikh Hadoop Installation Subject: Big Data Analytics
2020-2021
</configuration>

14-3

OPEN (/usr/local/hadoop/etc/hadoop/mapred-site.xml)
type and save

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
Faculty: Ms. Sana Shaikh Hadoop Installation Subject: Big Data Analytics
2020-2021

14-4 OPEN

(/usr/local/hadoop/etc/hadoop/yarn-site.xml)
type and save

<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>

15.

$ hdfs namenode -format


Faculty: Ms. Sana Shaikh Hadoop Installation Subject: Big Data Analytics
2020-2021

Now to start hadoop you can use two command

$ start-dfs.sh
$ start-yarn.sh

or you can also use deprecated command as

$ start-all.sh

To check the which components are working you can use bellow command
$ jps

you will get output as

type in browser:
https://localhost:50070/
Faculty: Ms. Sana Shaikh Hadoop Installation Subject: Big Data Analytics
2020-2021

You might also like