0% found this document useful (0 votes)
114 views28 pages

Installation Guide - Hadoop-2.6.0

The document provides step-by-step instructions for installing Hadoop 2.6.0 on an Ubuntu 14.04 virtual machine. It details downloading and configuring the VMware software, Ubuntu image, and Java. It then guides the user through extracting, configuring, and running Hadoop including editing configuration files, formatting the namenode, starting services, and checking the Hadoop filesystem. The goal is to set up a fully functioning Hadoop cluster on a single node for testing and learning purposes.

Uploaded by

Suresh Sai
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
114 views28 pages

Installation Guide - Hadoop-2.6.0

The document provides step-by-step instructions for installing Hadoop 2.6.0 on an Ubuntu 14.04 virtual machine. It details downloading and configuring the VMware software, Ubuntu image, and Java. It then guides the user through extracting, configuring, and running Hadoop including editing configuration files, formatting the namenode, starting services, and checking the Hadoop filesystem. The goal is to set up a fully functioning Hadoop cluster on a single node for testing and learning purposes.

Uploaded by

Suresh Sai
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 28

Big Data and Hadoop

HADOOP 2.6.0 INSTALLATION GUIDE

Step 1: Download VMware from this link:


https://my.vmware.com/web/vmware/downloads

Step 2: Download the Ubuntu 14.04 image:


http://www.traffictool.net/vmware/ubuntu1404t.html

www.skillpeed.com Page 1
Step 3: Extract the file in a location on your machine

Step 4: Open VM and click on open a virtual machine

Step 5: Now add your extract: Ubuntu

Now your VMware would look like the image given below. Click on edit virtual machine setting

www.skillpeed.com Page 2
Step 6: Increase your VMware memory min (6 GB), then click ok.

www.skillpeed.com Page 3
Now click on play virtual machine

Youll see the console like:

www.skillpeed.com Page 4
Step 7: Click on Terminal

www.skillpeed.com Page 5
Step 8: Install Java and add the repository as shown in screenshot below:

Step 9: It will ask for password, enter password

And then press enter again as show below:

www.skillpeed.com Page 6
Step 10: Now update your system

It will take some minutes to complete. After completion the screen will look below:

www.skillpeed.com Page 7
www.skillpeed.com Page 8
Step 11: Now invoke the Java-7-installer as shown below:

It will ask for accepting the license. Hit OK as shown

www.skillpeed.com Page 9
Step 12: Click on Yes

www.skillpeed.com Page 10
Step 13: Java installation will continue for some time. Once it is complete, the screen
would look like the following:

Step 14: Confirm Java version as shown below:

www.skillpeed.com Page 11
Step 15: Now install openssh server

Step 16: Generate ssh keys

www.skillpeed.com Page 12
Step 17: Just press enter for password, dont enter any password by yourself. Once done it
will look like below:

Step 18: Enable ssh access

www.skillpeed.com Page 13
Step 19: Test ssh acces

Step 20: Now disable the IPv6 as shown below:

Open the sysctl.conf file as follows:

www.skillpeed.com Page 14
Step 21: Add the lines shown in screenshot below:

# IPv6 disabled

net.ipv6.conf.all.disable_ipv6 = 1

net.ipv6.conf.default.disable_ipv6 = 1

net.ipv6.conf.lo.disable_ipv6 = 1

Step 22: Next reboot your virtual machinen

Step 23: Once you restart the machine test if IPv6 is disabled, your output should be as
shown in the screenshot below:

www.skillpeed.com Page 15
Step 24: Now we are ready for Hadoop Installation

Download Hadoop-2.6.0 from this link:

http://supergsego.com/apache/hadoop/common/hadoop-2.6.0/

Step 25: Now extract the tar file as shown below

www.skillpeed.com Page 16
Step 26: You would see Hadoop folder as follows

Step 27: Open your .basrc file as shown below:

Step 28: Add the highlighted lines as shown below:

www.skillpeed.com Page 17
# -- HADOOP ENVIRONMENT VARIABLES START -- #

export JAVA_HOME=/usr/lib/jvm/java-7-oracle

export HADOOP_HOME=/home/user/hadoop-2.6.0

export PATH=$PATH:$HADOOP_HOME/bin

export PATH=$PATH:$HADOOP_HOME/sbin

export HADOOP_MAPRED_HOME=$HADOOP_HOME

export HADOOP_COMMON_HOME=$HADOOP_HOME

export HADOOP_HDFS_HOME=$HADOOP_HOME

export YARN_HOME=$HADOOP_HOME

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

# -- HADOOP ENVIRONMENT VARIABLES END -- #

Step 29: You should now close the terminal and open a new terminal to check Hadoop
and Java homes as shown below: Write command

echo $JAVA_HOME

echo $HADOOP_INSTALL

Step 30: Now we will configure Hadoop. Go to hadoop-2.6.0/etc/hadoop in Hadoop folder


(Command :- cd ~/hadoop-2.6.0/etc/hadoop/), you should be able see all the config
files as shown below:

www.skillpeed.com Page 18
Step 31: Add the highlighted lines as shown below:

export JAVA_HOME=/usr/lib/jvm/java-7-oracle

Step 32:Open edit core-site.xml and add the highlighted lines as shown below:

www.skillpeed.com Page 19
Step 33: Edit core-site.xml as followed and add the highlighted lines as shown below:

<configuration>

<property>

<name>fs.default.name</name>

<value>hdfs://localhost:9000</value>

</property>

</configuration>

www.skillpeed.com Page 20
Step 34: Open yarn-sit.xml

Step 35: Edit yarn-site.xml as follows and add the highlighted lines as shown below:

<configuration>

<!-- Site specific YARN configuration properties -->

<property>

<name>yarn.nodemanager.aux-services</name>

www.skillpeed.com Page 21
<value>mapreduce_shuffle</value>

</property>

<property>

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

</configuration>

To open mapred-site.xml take this step

(Command :- sudo cp mapred-site.xml.templete mapred-site.xml)

www.skillpeed.com Page 22
Step 36: Open mapred-site.xml

Step 37: Add the highlighted lines as shown below:

<configuration>

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

www.skillpeed.com Page 23
</configuration>

Step 38: For hdfs - Create two directories for Hadoop storage

Commands:- sudo mkdir -p hadoop_tmp/hdfs/namenode

sudo mkdir -p hadoop_tmp/hdfs/datanode

Step 39: Open hdfs-site.xml

www.skillpeed.com Page 24
Step 40: Add the highlighted lines as shown below:

<configuration>

<property>

<name>dfs.replication</name>

<value>1</value>

</property>

<property>

<name>dfs.namenode.name.dir</name>

<value>file:/home/user/hadoop_tmp/hdfs/namenode</value>

</property>

<property>

<name>dfs.datanode.data.dir</name>

www.skillpeed.com Page 25
<value>file:/home/user/hadoop_tmp/hdfs/datanode</value>

</property>

</configuration>

Change folder permission

Write this command

sudo chown -R user:user /home/user/hadoop-2.6.0/etc/hadoop

sudo chown -R user:user /home/user/hadoop_tmp

Step 41: Format namenode:( Command: - hdfs namenode -format )

Step 42: Start all the services of Hadoop as shown below:

www.skillpeed.com Page 26
Step 43: You can confirm the services by running jps command as shown below:

STEP 44 : Type command to check the directory of hdfs

Command:- hadoop fs -ls / (If nothing exists here then)

Command:- hadoop fs -mkdir /user

OR you can see the directories through the web browser.

Link :- localhost:50070

Go to Utilities Click Browse the file system.

Here all hadoop hdfs files in your system.

www.skillpeed.com Page 27
Link :- http://localhost:50070/dfshealth.jsp

Eclipse : sudo wget http://eclipse.stu.edu.tw/technology/epp/downloads/release/mars/1/eclipse-jee-mars-1-linux-gtk.tar.gz

www.skillpeed.com Page 28

You might also like