Upto 20% Scholarship on Live Online Classes

- Technology Blogs -

Hadoop HDFS Commands Cheat Sheet

Hadoop HDFS Commands Cheat Sheet

Welcome to the new article for learning Hadoop. In this article, we will talk about HDFS commands. While learning Hadoop you must have realized HDFS is core component of Hadoop ecosystem. It is distributed files system meant to store file large files. So just like any other file system it also exposes command line interface to interact with. Using these commands, we can read, write, delete files and directory. HDFS commands are very much identical to Unix FS commands. Newer of versions of hadoop comes preloaded with support for many other file systems like HFTP FS, S3 FS.

All HDFS commands take resource path as arguments. The full path format is “scheme://authority/path”, where “scheme” keyword is filesystem identifier, For HDFS the scheme is ‘HDFS’, and for the Local Unix file system it is ‘file’. The scheme and authority parameters are optional, when not provided default scheme specified in the core-site.xml is used. A full HDFS path for any file or directory like /user/hadoop/myDir1 can be specified as hdfs://namenodehost//user/hadoop/myDir1 or simply as /user/hadoop/myDir1. To keep things simple, this article mainly focuses on HDFS filesystem.

In our journey of Hadoop commands, very first and useful command is ‘help’. This command display help for other commands or list of commands available in the Hadoop shell along with usage guide. If you ever get confuse about any command’s syntax ‘help’ command is quickest and most authentic way to go. Let’s have a look at it.

Help Commands:  Access Hadoop Command Manual
hadoop Help Commands
Now we learned about help command, let’s move to other commands. This article categorizes HDFS commands into 2 categories on the basis of their usage. First try to master “mostly used command” section these set of commands will help you get most of your work done.

Hadoop used commands

ls: It is used for listing files and directories on HDFS

Usage: hdfs dfs -ls [-R] <args>
Example: hdfs dfs -ls /user/hadoop/myDir
Optional: -R argument is used to list directories recursively.

Hadoop used commands

copyToLocal: as name suggests, it is used for copying file(s) from HDFS file system to local Unix filesystem.

Read:   HBase Interview Questions And Answers

This command works only with files NOT with directories.

Usage: hdfs dfs -copyToLocal <hdfsPath> <localUnixPath>
Example: hdfs dfs -copyToLocal /user/hadoop/myDir/myHadoopFile /home/myuser/mylocalFile

copyFromLocal: as name suggests, it is used for copying file from local Unix filesystem to HDFS file system.

This command works only with files NOT with directories.

Usage: hdfs dfs -copyFromLocal <localUnixPath> <hdfsPath>
Example: hdfs dfs -copyFromLocal /home/myuser/myFile /user/hadoop/myDir/myFile
Optional: -f argument will overwrite the destination if it already exists.

cat: it is used for displaying content of HDFS file on console

Usage: hdfs dfs -cat <hdfsPath> [otherhdfsPath …]
Example:  hdfs dfs -cat /user/hadoop/file1
hdfs dfs -cat /user/hadoop/file2 /user/hadoop/file3

cp: it is used for copying files/directories from one HDFS location to another HDFS location

Usage: hdfs dfs -cp [-f] <srcHdfsPath> <destHdfsPath>
Example: hdfs dfs -cp /user/hadoop/file1 /user/hadoop/file2
Optional: -f argument will overwrite the destination if it already exists.

get: This command is used to get/download files/directories from HDFS to local file system. This is similar to ‘copyFromLocal’ command, except it works with files as well as directories whereas ‘copyFromLocal’ only works with file.

Usage: hdfs dfs -get <srcHdfsPath> <localUnixDst>
Example:  hdfs dfs -get /user/hadoop/file /home/user/localfile

put: this command is counterpart of get command, i.e. it used to copy files/directories from local filesystem to HDFS. And it is similar to ‘copyToLocal’ but works with files and directories together

Usage: hdfs dfs -put <localUnixPath> ... <dstHdfsPath>
Example:  hdfs dfs -put localfile /user/hadoop/hadoopfile
hdfs dfs -put localfile1 localfile2 /user/hadoop/myDir

mkdir: it is used for creating directory/directories in HDFS

Usage: hdfs dfs -mkdir [-p] <hdfsPaths>
Example:  hdfs dfs -mkdir /user/hadoop/myHadoopDir
Optional: -p argument creates parent directories along the path

mv: it is used for moving file/directory from one HDFS location to another HDFS location

Usage: hdfs dfs -mv <sourceHdfspath> <destHadoopPath>
Example: hdfs dfs -mv /user/hadoop/file1 /user/hadoop/file2

rm: it is used for deleting file from HDFS location. Once file is deleted from its original HDFS location it goes into ‘trash’ HDFS directory.

Usage: hdfs dfs -rm [-f] [-r|-R] [-skipTrash] <hdfsPath>
Example: hdfs dfs -rm /user/hadoop/myHadoopFile
Optional: -f argument will force delete without any warning
-R argument deletes the directory and any file/directory under it recursively.
-r option is equivalent to -R.
-skipTrash option will skip trash directory, and delete the file(s) immediately

rmr: This command is similar to ‘rm’ command,the only difference is that it works with directories.

Usage: hdfs dfs -rmr [-skipTrash] <hdfsDirPath>
Example: hdfs dfs -rmr /user/hadoop/myDir

tail: this command is used for displaying last few lines of HDFS file on console.

Usage: hdfs dfs -tail [-f] <hdfsFilePath>
Example: hdfs dfs -tail /user/hadoop/myHadoopFile

Hadoop useful commands

appendToFile: it is used for appending content of local file(s) into one hdfs file.

Usage hdfs dfs -appendToFile <localSourcePath> <hdfsFilePath>
Example:  hdfs dfs -appendToFile mylocalfile /user/hadoop/myhadoopfile

chmod: is used for changing permission of hdfs file/directory.

Usage: hdfs dfs -chmod [-R] <MODE> <hdfsPath>
Example: hdfs dfs -chmod 777 /user/hadoop/myHadoopFile
Optional: -R argument make the change recursively

chown: it is used for changing owner of hdfs file/directory

Usage:  hdfs dfs -chown [-R] [OWNER][:[GROUP]] <hdfsPath>
Example: hdfs dfs -chown devUser:developers /user/hadoop/myHadoopPath
Optional: -R argument make the change recursively

count: it is used to count number of directories and files inside a directory

Usage:  hdfs dfs -count [-q] [-h] <hdfsPath>
The output columns with count are: DIR_COUNT, FILE_COUNT, CONTENT_SIZE FILE_NAME
Example: hdfs dfs -count /user/hadoop/myHadoopFile
Optional: -h argument shows sizes in human readable format e.g. 64.0m instead of 67108864

du: it is used to displays sizes of files and directories present in the specified directory.

Usage:  hdfs dfs -du [-s] [-h] <hdfsPath>
Example: hdfs dfs -du /user/hadoop/myHadoopFile
Optional:  -s option will result in an aggregate summary of file lengths being displayed, rather than the individual files
-h option will format file sizes in a "human-readable" fashion (e.g 64.0m instead of 67108864)

Hadoop useful commands

getmerge: it used to download/get multiple files from one hdfs directory as a single consolidated file onto local filesystem

Usage:  hdfs dfs -getmerge <hdfsDirPath> <localdst>
Example:  hdfs dfs -getmerge /user/hadoop/myHadoopDir/* /home/user/singleLocalFile

moveFromLocal: this command is similar to ‘copyFromLocal’ except it deletes local file after copying to hdfs.

Usage:  hdfs dfs -moveFromLocal <localsrcPath> <hdfsFilePath>
Example:  hdfs dfs -moveFromLocal mylocalFile /user/hadoop/myHadoopFile

moveToLocal: it is similar to ‘copyToLocal’ command except it deletes original hdfs file after copying it to local file system.

Usage: hdfs dfs -moveToLocal <hdfsFilePath> <localDestpath>
Example: hdfs dfs -moveToLocal /user/hadoop/myHadoopFile mylocalFile

setrep: it is used for changing replication level of a hdfs file/directory

Usage:  hdfs dfs -setrep [-w] <numReplicas> <path>
Example: hdfs dfs -setrep -w 3 /user/hadoop/dir1
Optional:  -w flag force command to wait for the replication to complete.

stat: it is used to show stats about hdfs file/directory

Usage:  hdfs dfs -stat <hdfsFilePath>
Example:  hdfs dfs -stat /user/hadoop/myHadoopFile

text:  it is more powerful version of ‘cat’ command. cat command can display content if specified hdfs

Read:   What is Spark? Apache Spark Tutorials Guide for Beginner

file is text file but ‘text’ command can display content from compressed gzip files also.

Usage: hdfs dfs -text <hdfsFilePath>
Example: hdfs dfs -text /user/hadoop/myHadoopFile.gzip

touchz: it is used to create empty file on hdfs file system

Usage: hdfs dfs -touchz <hdfsFilePath>
Example: hdfs dfs -touchz /user/hadoop/myHadoopFile

fsck: this command is used for checking health of HDFS, it is designed for reporting problems with various files, for example, missing blocks for a file or under-replicated blocks.

Usage: hadoop fsck <hdfsFilePath> [-move | -delete | -openforwrite]
Example: hdfs dfs - hdfsFilePath /user/hadoop/myHadoopFile -delete

Conclusion:

We just walked over some HDFS commands which are used on day to day basis while working with hadoop. We learned how to perform basic file system operations and work with files and directories present in HDFS. We also understood how to get files from HDFS to our local file system and other way around. Besides basic filesystem operation we explored a few advanced features of HDFS data management using the command line.

JanBask Training

JanBask Training

JanBask Training is a leading Global Online Training Provider through Live Sessions. The Live classes provide a blended approach of hands on experience along with theoretical knowledge which is driven by certified professionals.



Write a Comment

avatar
  Subscribe  
Notify of

Trending

Top 30 Core Java Interview Questions and Answers for Fresher, Experienced Developer
What Is Trigger In Salesforce?
Top 30 Manual Testing Interview Questions & Answers for Fresher
Different Types of SQL Server & SQL Database Functions
Top 20 AWS Interview Question and Answers For Fresher, Experienced Developer

Related Posts

Your Complete Guide to Apache Hive Data Models
CCA Spark & Hadoop Developer Certification Exam Practice Tests
What is Flume? Apache Flume Tutorial Guide For Beginners
Kafka Interview Questions and Answers
An Introduction to Apache Spark and Spark SQL