Scribd
Upload
14 views

2.5 - Demo-Hadoop Install - Java SSH Configure

Hadoop configuration

Uploaded by

Alexandra Berger
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
14 views

2.5 - Demo-Hadoop Install - Java SSH Configure

Hadoop configuration

Uploaded by

Alexandra Berger
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 16

CBTU presents a course on Big data and Hadoop

Module 2:
Section 2.5: Hadoop install –
Configure Java and SSH

Apache and the Apache feather logo are trademarks of The Apache Software Foundation, Licensed under the Apache License, Version 2.0.
All the logos, trademarks are copyright of the respective companies.
Hadoop installation
• Hadoop for Linux distro
– Download Hadoop stable release
– Verify integrity downloaded Hadoop release
• Prerequisites
– OS (GNU/Linux, Windows)
– Java
– SSH
Prerequisites – Java
Hadoop require:
• Java™
Java-1.8.0-OpenJDK http://openjdk.java.net/
• Recommended Java versions for Hadoop:
– http://wiki.apache.org/hadoop/HadoopJavaVersions
Verify Java installed status
Java is the main prerequisite for Hadoop. Java is
generally installed in all Linux systems.
Check the Java installed version
– java –version
Configure: hadoop-env.sh
– export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk
– export HADOOP_HOME=/home/ken/k-soft/hadoop-3.1.0
– export HADOOP_MAPRED_HOME=$HADOOP_HOME
– export HADOOP_COMMON_HOME=$HADOOP_HOME
– export HADOOP_HDFS_HOME=$HADOOP_HOME
– export YARN_HOME=$HADOOP_HOME
– export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
– export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
– export HADOOP_INSTALL=$HADOOP_HOME
– export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
– export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
– export HADOOP_PREFIX=$HADOOP_HOME
– export HADOOP_LIBEXEC_DIR=$HADOOP_HOME/libexec
– export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native:$JAVA_LIBRARY_PATH
– export HADOOP_YARN_HOME=$HADOOP_HOME
Check Hadoop installation
• By default, Hadoop is configured to run in a
standalone, non-distributed mode, as a single Java
process. This is useful for debugging.
SSH localhost
[root@localhost hadoop-3.1.0]# sbin/start-dfs.sh
Starting namenodes on [localhost.localdomain]
Last login: Tue May 22 15:54:45 IST 2018 on pts/0
localhost.localdomain: Warning: Permanently added
'localhost.localdomain' (ECDSA) to the list of known
hosts.
localhost.localdomain: Permission denied
(publickey,gssapi-keyex,gssapi-with-mic,password).
ssh localhost without password
# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
# cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
# chmod 0600 ~/.ssh/authorized_keys
# ssh localhost
Other commands for troubleshooting
# ls –la
# ssh -v localhost
sshd_config
– Configure PasswordAuthentication no
Ready to run Hadoop
• Start hdfs namenode
• sbin/start-dfs.sh
• Hadoop access ports
• Troubleshooting
• Hadoop run
• sbin/stop-dfs.sh

You might also like