In this article, the basic syntax of hadoop file system i.e HDFS has been explained with examples and screen shot. This is very useful for the beginners who are interested to explore the big data world and HDFS is the gate to that world.
Hadoop is open source software [It is a java frame work] which runs on a cluster of commodity hardware machines. It provides both storage [HDFS] and processing [MAP REDUCE] in distributed manner. It has capable of processing huge volume of data that is ranging from Giga bytes to Peta bytes.
HDFS Commands
hdfs dfs
hadood fs:
hdfs dfs/hadoop fs
1. Creating a directory [HDFS]
Syntax:
- hdfs dfs –mkdir <Directory name along with path details >
Example:
- hdfs dfs –mkdir /user/root/hadoop_mahendhar
Screenshot
2. Listing the contains of the hadoop directory
Syntax:
- hdfs dfs –ls < argument like absolute path of the file >
Example:
- hdfs dfs –ls /user/root/hadoop_mahendhar
Screen Shot:
3. Create a file in local file system and put the file in HDFS
Create a file in local system by vi <file_name>, add some texts and save, exit.
Syntax: vi First_hadoop.txt
Putting the normal file to hadoop file system
Syntax:
- hdfs dfs –put <Local file path with file name > <hadoop destination path with file name >
Example:
- hdfs dfs –put First_hadoop.txt /user/root/hadoop_mahendhar
Screen Shot:
4. Moving a normal file to hadoop file system
Syntax:
- hdfs dfs – moveFromLocal <Local file path with file name > <hadoop destination path with file name >
Example:
- hdfs dfs –moveFromLocal /root/Second_hadoop.txt /user/root/hadoop_mahendhar
Screenshot:
Note:
1. Before executing the above command, ensure that the second_hadoop.txt file is created in the Local normal file system.
2. This operation will move the local file, so there is no local copy of the file exist after this operation.
5. For listing all directories and sub-directories recursively
Syntax:
- hdfs dfs –lsr <hadoop directory>
Example:
- hdfs dfs -lsr /user/root/hadoop_mahendhar/
- Note: Create more directory and sub directory to validate this command correctly.
6. Check the size of the file in HDFS
Syntax:
- hdfs dfs – du <File path with file name >
Example:
- hdfs dfs – du /user/root/hadoop_mahendhar/
Screen Shot:
7. Download a file from HDFS to normal file system
Syntax:
- hdfs dfs – get <hadoop file path details with file name > < local file path details with file name>
Example:
- hdfs dfs – get /user/root/hadoop_mahendhar/Second_hadoop.txt /root/local_files/
Screen Shot:
8. Getting a directory of files from HDFS and merge into a single file in normal file system
Syntax:
- hdfs dfs – getmerge <HDFS file directory > < Local file path with file details > < add new line>
Example:
- hdfs dfs – getmerge /user/root/hadoop_mahendhar/ /root/local_files/hadoop_merge_file.txt
Screen Shot:
Note:
1. The add newline is optional and it will just add a new line at the end of the each file.
2. Before this make sure you have created 2-3 files in HDFS so that you can check and validate the file contain with normal file w.r.t size
9. Copying data from one node to another node in HDFS
Syntax:
- hdfs dfs – distcp <node1 file path details > <node 2 file path details >
10. Display the contain of the data
Syntax:
- hdfs dfs – cat < File path details with file name >
Example:
- hdfs dfs –cat /user/root/hadoop_mahendhar/ First_hadoop.txt
Screen Shot:
11. Change the group, owner, permission of the file or directory
Syntax:
- hdfs dfs –chgrp [-R] <New group Name > <File or directory >
- hdfs dfs –chmod [ -R] <Fileor directory>
- hdfs dfs –chown <New Owner name> < File or directory>
12. Copying and Moving files within HDFS
Syntax:
- hdfs dfs –cp <First file path details > < Destination file details >
- hdfs dfs –mv <First file path details > < Destination file details >
13. Empty the hadoop thrash
Syntax:
- hdfs dfs –expunge
Screen Shot:
Apache Hadoop Terms/Abbreviations click here
