Scribd
Upload
47 views

Setting Hadoop and Mysql 8.0

This document provides instructions to install Hadoop on an Ubuntu operating system in 11 steps: 1. Install Java JDK 8 and configure environment variables for Java home. 2. Download and extract the Hadoop tar file. 3. Configure Hadoop configuration files like core-site.xml, hdfs-site.xml and yarn-site.xml. 4. Format the HDFS file system and start all Hadoop daemons using start-all.sh.

Uploaded by

21020279 Trần Diệu Anh
Copyright
© © All Rights Reserved
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
47 views

Setting Hadoop and Mysql 8.0

This document provides instructions to install Hadoop on an Ubuntu operating system in 11 steps: 1. Install Java JDK 8 and configure environment variables for Java home. 2. Download and extract the Hadoop tar file. 3. Configure Hadoop configuration files like core-site.xml, hdfs-site.xml and yarn-site.xml. 4. Format the HDFS file system and start all Hadoop daemons using start-all.sh.

Uploaded by

21020279 Trần Diệu Anh
Copyright
© © All Rights Reserved
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
You are on page 1/ 3

https://codewitharjun-medium-com.translate.

goog/install-hadoop-on-ubuntu-operating-
system-6e0ca4ef9689?source=author_recirc-----8ef3d665e331----0---------------------
6d0ea0b1_45d1_4004_b330_964496b9d0d0-------
&_x_tr_sl=en&_x_tr_tl=vi&_x_tr_hl=vi&_x_tr_pto=sc

1.Install java jdk 8

✍️ sudo apt install openjdk-8-jdk

- To check it’s there cd /usr/lib/jvm

2. open .bashrc file and paste these commands

✍️ sudo nano .bashrc

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export PATH=$PATH:/usr/lib/jvm/java-8-openjdk-amd64/bin
export HADOOP_HOME=~/hadoop-3.3.5/
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
export HADOOP_STREAMING=$HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming-
3.3.5.jar
export HADOOP_LOG_DIR=$HADOOP_HOME/logs
export PDSH_RCMD_TYPE=ssh

3. install ssh

✍️ sudo apt-get install ssh

4. Go to hadoop.apache.org website download the tar file


(hadoop.apache.org — download tar file of hadoop.)

✍️ tar -zxvf ~/Downloads/hadoop-3.3.5.tar.gz


(Extract the tar file)

✍️ cd hadoop-3.3.5/etc/hadoop

- now open hadoop-env.sh


✍️ sudo nano hadoop-env.sh

JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 (set the path for JAVA_HOME)

5. Open core-site.xml
✍️ sudo nano core-site.xml //// gedit core-site.xml

<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value> </property>
<property>
<name>hadoop.proxyuser.dataflair.groups</name> <value>*</value>
</property>
<property>
<name>hadoop.proxyuser.dataflair.hosts</name> <value>*</value>
</property>
<property>
<name>hadoop.proxyuser.server.hosts</name> <value>*</value>
</property>
<property>
<name>hadoop.proxyuser.server.groups</name> <value>*</value>
</property>
</configuration>

6. Open hdfs-site.xml
✍️ sudo nano hdfs-site.xml //// gedit hdfs-site.xml

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>

<property>
<name>dfs.client.block.write.locateFollowingBlock.retries</name>
<value>10</value>
</property>

7. Open mapred-site.xml
✍️ sudo nano mapred-site.xml //// gedit mapred-site.xml

<configuration>
<property>
<name>mapreduce.framework.name</name> <value>yarn</value>
</property>
<property>
<name>mapreduce.application.classpath</name>

<value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/
hadoop/mapreduce/lib/*</value>
</property>
</configuration>

8. Open yarn-site.xml
✍️ sudo nano yarn-site.xml //// gedit yarn-site.xml

<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.env-whitelist</name>

<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREP
END_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
</property>
</configuration>

9. Start ssh
✍️ ssh localhost
✍️ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
✍️ chmod 0600 ~/.ssh/authorized_keys
✍️ hadoop-3.3.5/bin/hdfs namenode -format

10. format the file system

✍️ export PDSH_RCMD_TYPE=ssh

11. To start

✍️ start-all.sh
(Start NameNode daemon and DataNode daemon)

You might also like