Scribd
Upload
167 views

Admin Commands

This document provides a quick reference to common Hadoop administration commands organized into sections on: 1) Namenode commands, 2) fsck commands, 3) Job commands, 4) dfsadmin commands, 5) Safe Mode commands, 6) Configuration files, 7) mradmin commands, 8) Balancer commands, and 9) Filesystem commands. It lists the command, a brief description, and examples of usage for key Hadoop cluster management tasks.

Uploaded by

Katrina Camacho
Copyright
© © All Rights Reserved
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
167 views

Admin Commands

This document provides a quick reference to common Hadoop administration commands organized into sections on: 1) Namenode commands, 2) fsck commands, 3) Job commands, 4) dfsadmin commands, 5) Safe Mode commands, 6) Configuration files, 7) mradmin commands, 8) Balancer commands, and 9) Filesystem commands. It lists the command, a brief description, and examples of usage for key Hadoop cluster management tasks.

Uploaded by

Katrina Camacho
Copyright
© © All Rights Reserved
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
You are on page 1/ 6

Home

Free eBook

Start Here

Contact

About

Quick Apache Hadoop Admin Command Reference


Examples
by KARTHIKEYAN SADHASIVAM on FEBRUARY 18, 2015

If you are working on Hadoop, youll realize


there are several shell commands available to manage your hadoop cluster.
This article provides a quick handy reference to all Hadoop administration commands.
If you are new to big data, read the introduction to Hadoop article to understand the
basics.

1. Hadoop Namenode Commands


Command

Description

hadoop namenode -format


hadoop namenode -upgrade
start-dfs.sh
stop-dfs.sh
start-mapred.sh
stop-mapred.sh
hadoop namenode -recover

Format HDFS filesystem from Namenode


Upgrade the NameNode
Start HDFS Daemons
Stop HDFS Daemons
Start MapReduce Daemons
Stop MapReduce Daemons
Recover namenode metadata after a cluster failure

-force

(may lose data)

2. Hadoop fsck Commands


Command

Description

hadoop fsck /
hadoop fsck / -files
hadoop fsck / -files -blocks
hadoop fsck / -files -blocks
-locations
hadoop fsck / -files -blocks
-locations -racks
hadoop fsck -delete

Filesystem check on HDFS


Display files during check
Display files and blocks during check
Display files, blocks and its location
during check
Display network topology for data-node
locations
Delete corrupted files
Move corrupted files to /lost+found
directory

hadoop fsck -move

3. Hadoop Job Commands


Command
hadoop job -submit <jobfile>
hadoop job -status <jobid>
hadoop job -list all
hadoop job -list-activetrackers

Description
Submit the job
Print job status completion percentage
List all jobs
List all available TaskTrackers

hadoop job -set-priority


<job-id> <priority>

Set priority for a job. Valid priorities:


VERY_HIGH, HIGH, NORMAL, LOW,
VERY_LOW

hadoop job -kill-task


<task-id>

Kill a task

hadoop job -history

Display job history including job details, failed and


killed jobs

4. Hadoop dfsadmin Commands

Command

Description

hadoop dfsadmin -report


hadoop dfsadmin
-metasave file.txt
hadoop dfsadmin
-setQuota 10 /quotatest
hadoop dfsadmin
-clrQuota /quotatest

Report filesystem info and statistics

hadoop dfsadmin
-refreshNodes
hadoop fs -count -q
/mydir
hadoop dfsadmin
-setSpaceQuota /mydir
100M
hadoop dfsadmin
-clrSpaceQuota /mydir
hadooop dfsadmin
-saveNameSpace

Save namenodes primary data structures to file.txt


Set Hadoop directory quota to only 10 files
Clear Hadoop directory quota
Read hosts and exclude files to update datanodes that
are allowed to connect to namenode. Mostly used to
commission or decommsion nodes
Check quota space on directory /mydir
Set quota to 100M on hdfs directory named /mydir
Clear quota on a HDFS directory
Backup Metadata (fsimage & edits). Put cluster in
safe mode before this command.

5. Hadoop Safe Mode (Maintenance Mode) Commands


The following dfsadmin commands helps the cluster to enter or leave safe mode, which
is also called as maintenance mode. In this mode, Namenode does not accept any
changes to the name space, it does not replicate or delete blocks.

Command
hadoop dfsadmin -safemode
enter
hadoop dfsadmin -safemode
leave
hadoop dfsadmin -safemode get

Description
Enter safe mode
Leave safe mode

Get the status of mode


Wait until HDFS finishes data block
hadoop dfsadmin -safemode wait
replication

6. Hadoop Configuration Files


File

Description

hadoop-env.sh
core-site.xml
hdfs-site.xml
mapred-site.xml
masters
slaves

Sets ENV variables for Hadoop


Parameters for entire Hadoop cluster
Parameters for HDFS and its clients
Parameters for MapReduce and its clients
Host machines for secondary Namenode
List of slave hosts

7. Hadoop mradmin Commands


Command

Description

hadoop mradmin -safemode get


hadoop mradmin -refreshQueues
hadoop mradmin -refreshNodes

Check Job tracker status


Reload mapreduce configuration
Reload active TaskTrackers
Force Jobtracker to reload service
ACL
Force jobtracker to reload user
group mappings

hadoop mradmin -refreshServiceAcl


hadoop mradmin
-refreshUserToGroupsMappings

8. Hadoop Balancer Commands


Command

Description

start-balancer.sh
hadoop dfsadmin -setBalancerBandwidth
<bandwidthinbytes>

Balance the cluster


Adjust bandwidth used by the
balancer
Limit balancing to only 20%
resources in the cluster

hadoop balancer -threshold 20

9. Hadoop Filesystem Commands


Command

Description

hadoop fs -mkdir mydir


hadoop fs -ls

Create a directory (mydir) in HDFS


List files and directories in HDFS

hadoop fs -cat myfile


hadoop fs -du
hadoop fs -expunge
hadoop fs -chgrp hadoop file1
hadoop fs -chown huser file1
hadoop fs -rm file1
hadoop fs -touchz file2
hadoop fs -stat file1
hadoop fs -test -e file1
hadoop fs -test -z file1
hadoop fs -test -d file1

View a file content


Check disk space usage in HDFS
Empty trash on HDFS
Change group membership of a file
Change file ownership
Delete a file in HDFS
Create an empty file
Check the status of a file
Check if file exists on HDFS
Check if file is empty on HDFS
Check if file1 is a directory on HDFS

10. Additional Hadoop Filesystem Commands


Command

Description

hadoop fs -copyFromLocal <source>


<destination>

Copy from local fileystem to


HDFS
e.g: Copies file1 from local FS
to data dir in HDFS
copy from hdfs to local
filesystem
e.g: Copies file1 from HDFS
data directory to /var/tmp on
local FS
Copy from remote location to
HDFS
Copy from HDFS to remote
directory
Copy data from one cluster to
another using the cluster URL
Move data file from the local
directory to HDFS
Set the replication factor for
file1 to 3
Merge files in mydir directory
and download it as one big file

hadoop fs -copyFromLocal file1 data


hadoop fs -copyToLocal <source>
<destination>
hadoop fs -copyToLocal data/file1 /var/tmp
hadoop fs -put <source> <destination>
hadoop fs -get <source> <destination>
hadoop distcp hdfs://192.168.0.8:8020/input
hdfs://192.168.0.8:8020/output
hadoop fs -mv file:///data/datafile
/user/hduser/data
hadoop fs -setrep -w 3 file1
hadoop fs -getmerge mydir bigfile

You might also like