Scribd
Upload
11 views3 pages

Hadoop Fs

Uploaded by

sunilbjoshi123
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
11 views3 pages

Hadoop Fs

Uploaded by

sunilbjoshi123
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 3

Installation:

Install java8 > C:\Java\jdk-1.8


ENV : user : JAVA_HOME C:\Java\jdk-1.8
System : Path : edit : C:\Java\jdk-1.8\bin
Extract Hadoop tar file
Hadoop/etc : Hadoop.env -> JAVA_Home= C:\Java\jdk-1.8
Core-site.xml (core configuration settings for Hadoop), hdfs-site.xml (HDFS specific
configuration) , mapred.xml (mapreduce - yarn), yarn-site.xml,
User: HADOOP_HOME C:\hadoop\bin
System:add C:\hadoop\bin & C:\hadoop\sbin to path
Run Cmd as administrator
hdfs namenode -format
jps
start-all.cmd
jps

Hadoop fs
[any file system, any path, local fs, cloud fs as well]
OR hdfs dfs
[Hadoop fs]
Commands:
List : hadoop fs -ls /
Create a path : hdfs dfs -mkdir -p /user/SunilJoshi
make a directory : hadoop fs -mkdir /input
create a file through cmd : hadoop fs -touchz /input/ap.txt
copy from local :
create a text file : D/MSSU/file.txt -> write text
hadoop fs -put D:\MSSU\file.txt /input OR hadoop fs -copyFromLocal D:\MSSU\file.txt /input
Show contents of a file on CMD:
hadoop fs -cat /folder1/f11.txt
Copy to local:
hadoop fs -copyToLocal /folder1/f11.txt D:\MSSU\fld\
Transfer one file from HDFS directory to another :
hadoop fs -cp /folder1/f11.txt /input/
hadoop fs -ls /input
hadoop fs -cat /input/f11.txt
move hdfs directory to another:
hadoop fs -mkdir /input1
hadoop fs -mv /folder1/f11.txt /input1/
Check whether file has moved or not : hadoop fs -ls /folder1
Disk usage: hadoop fs -du /input
Each and every file size

DUS:
It will display total size
hadoop fs -dus /input
test Command: (conditional checking) -d -e -f -s -r -w -z (multiple parameters)
-d : given path is directory or file
-e : whether the destination is exist or not
-f : file or not
-z : it will check whether the file is zero byte file or non-zero byte file
Hadoop fs -test -d /dd
Echo $?
Hadoop fs -test -d /folder1
Echo $?
Movefromlocal :
Hadoop fs -moveFromLocal a2.txt /folder1
Getmerge command:
hadoop fs -getmerge -nl /folder1/xyz.txt /folder1/xyz1.txt D:\MSSU\fld\merged.txt
appendtofile command:
create 2 files : a.txt & b.txt with some text
create empty file : hadoop fs -touchz c.txt
hadoop fs -appendToFile a.txt b.txt c.txt
check : hadoop fs -cat c.txt
checksum command : check integrity of a file .. whether file is modified or not .. checks the
hash value
hadoop fs -checksum /folder1/a.txt
want to find out whether the HDFS is running fine or not : status
hadoop fsck - /
erasure : https://data-flair.training/blogs/hadoop-hdfs-erasure-
coding/#:~:text=Advantages%20of%20HDFS%20Erasure%20Coding%20in%20Hadoop&tex
t=It%20also%20eliminates%20the%20need,Erasure%20Coding%20is%20only%2050%25.

You might also like