Delete files older than 30days on HDFS

This post helps in cleanup of HDFS files older than a certain date(30days) using a shell script.

 #!/bin/sh  
 #finding HDFS load time of particular folder  
 today=`date +'%s'`  
 hdfs dfs -ls /file/Path/ | grep "^d" | while read line ; do  
 dir_date=$(echo ${line} | awk '{print $6}')  
 difference=$(( ( ${today} - $(date -d ${dir_date} +%s) ) / ( 24*60*60 ) ))  
 filePath=$(echo ${line} | awk '{print $8}')  
 if [ ${difference} -gt 30 ]; then  
   hdfs dfs -rm -r $filePath  
 fi  
 done  

If you are facing any problems in deleting files, then please comment here.

How to Change the priority of Hadoop Jobs (Hive,MapReduce,pig)?

Hadoop provides a command line utility for changing the priority of running job. There are total 5 different priority level are there. These priority levels are listed below:

VERY_HIGH
HIGH
NORMAL
LOW
VERY_LOW

We can change the priority of the running hadoop job syntax is shown below:

 hadoop job -set-priority <job-id> <priority>  

Example:

 hadoop job -set-priority job_20170111540_64444 VERY_HIGH  

From the names of the priority levels, it is quite obvious that highest priority is given to the jobs whose priority level is VERY_HIGH and least priority is given to the jobs whose priority level is VERY_LOW.

Priority in Map Reduce Program:

In order to set priority for a job on Hadoop 1 cluster, you can use the following example:

 -> SET mapred.job.priority=<priority_value>;  

In order to set priority for a job on Hadoop 2 cluster, you can use the following example:

 -> SET mapreduce.job.priority=<priority_value>;  

Priority in Pig Program:

The below property is used to set the job priority is Pig Programming.

 grunt> SET job.priority 'high'  

The following <priority_value> values are supported:
very_low, low, normal, high, very_high

Priority for Hive Query:

In order to set priority for a HIVE query on Hadoop 1 cluster, you can use the following example:

 hive> SET mapred.job.priority=VERY_HIGH;  

In order to set priority for a HIVE query on Hadoop 2 cluster, you can use the following example:

 hive> SET mapreduce.job.priority=VERY_HIGH;  

If you are facing any problems in changing the priority of the hadoop jobs, then please comment here.

DISTRIBUTED CACHE IN HADOOP

Introduction :

In the Using third part jars and files in your MapReduce application(Distributed cache) entry i blogged about how to use Distributed Cache in Hadoop using DistributedCache API . But you can also have option of using command line option. You will have to use following steps to use DistributedCache programmatically In order to use it, first change your MapReduce Driver class to add job.addCacheFile()

Hadoop Map Reduce Project provides us this facility with something called as DistributedCache.
This Distributed Cache is configured with Job Configuration, What it does is, it provides read only data to all machine on the cluster.

Step 1 : Put file to HDFS

In order to use a file with DistributedCache API, it has to available on either hdfs:// or http:// URL, that is accessible to all the cluster members. So first step was to upload the file that you are interested in into HDFS, in my case i used following command to copy the /tmp/file1 file to hdfs.

 # hdfs dfs -put /tmp/file1 /cachefile1  

Step 2: Add cachefile in Job Configuration

Next step is to change the Driver class and add job.addCacheFile(new URI("/cachefile1")); call, this call takes the hdfs url of the file that you just uploaded to HDFS and passes it to DistributedCache class.

 Configuration conf = new Configuration();  
 Job job = new Job(conf, "wordcount");  
 DistributedCache.addCacheFile(new URI("/cachefile1"),job.getConfiguration());  

Step 3: Access Cached file

Now in your Mapper class you can read the file1 using normal File API

 Path[] cacheFiles = context.getLocalCacheFiles();  
 FileInputStream fileStream = new FileInputStream(cacheFiles[0].toString());  

You may like reading HADOOP directory structure Hadoop -1.0.4.tar.gz Directory Structure

If you know anyone who has started learning Hadoop and Java, why not help them out! Just share this post with them. Thanks for studying today!

Hortonworks or Cloudera Sandbox links

Cloudera QuickStart VM
Hortonworks Sandbox

HDFS Commands Reference

In this article, the basic syntax of hadoop file system i.e HDFS has been explained with examples and screen shot. This is very useful for the beginners who are interested to explore the big data world and HDFS is the gate to that world.

Hadoop is open source software [It is a java frame work] which runs on a cluster of commodity hardware machines. It provides both storage [HDFS] and processing [MAP REDUCE] in distributed manner. It has capable of processing huge volume of data that is ranging from Giga bytes to Peta bytes.

HDFS Commands

hdfs dfs

hadood fs:

hdfs dfs/hadoop fs

1.  Creating a directory [HDFS]

Syntax:

- hdfs dfs –mkdir <Directory name along with path details >

Example:

- hdfs dfs –mkdir /user/root/hadoop_mahendhar

Screenshot

2.  Listing the contains of the hadoop directory

Syntax:

- hdfs dfs –ls < argument like absolute path of the file >

Example:

- hdfs  dfs –ls /user/root/hadoop_mahendhar

Screen Shot:

3.  Create a file in local file system and put the file in HDFS

Create a file in local system by  vi <file_name>, add some texts and save, exit.

Syntax: vi First_hadoop.txt

Putting the normal file to hadoop file system

Syntax:

- hdfs dfs –put <Local file path with file name > <hadoop destination path with file name >

Example:

- hdfs dfs –put  First_hadoop.txt  /user/root/hadoop_mahendhar

Screen Shot:

4. Moving a normal file to hadoop file system 

Syntax: 

- hdfs dfs – moveFromLocal <Local file path with file name > <hadoop destination path with file name > 

Example:

- hdfs dfs  –moveFromLocal  /root/Second_hadoop.txt  /user/root/hadoop_mahendhar

Screenshot:

Note:

1. Before executing the above command, ensure that the second_hadoop.txt file is created in the Local normal file system. 

2. This operation will move the local file, so there is no local copy of the file exist after this operation. 

5. For listing all directories and sub-directories recursively 

Syntax:

- hdfs dfs –lsr <hadoop directory>

Example:

- hdfs  dfs -lsr /user/root/hadoop_mahendhar/ 

- Note: Create more directory and sub directory to validate this command correctly. 

6. Check the size of the file in HDFS 

Syntax: 

- hdfs dfs – du <File path with file name > 

Example:

- hdfs  dfs – du /user/root/hadoop_mahendhar/  

Screen Shot:

7. Download a file from HDFS to normal file system 

Syntax: 

- hdfs dfs – get <hadoop file path details with file name > < local file path details with file name> 

Example:

- hdfs dfs – get /user/root/hadoop_mahendhar/Second_hadoop.txt /root/local_files/ 

Screen Shot:

8. Getting a directory of files from HDFS and merge into a single file in normal file system 

Syntax: 

- hdfs dfs – getmerge <HDFS file directory > < Local file path with file details > < add new line> 

Example:

- hdfs dfs – getmerge /user/root/hadoop_mahendhar/ /root/local_files/hadoop_merge_file.txt 

Screen Shot:

Note:

1. The add newline is optional and it will just add a new line at the end of the each file. 

2. Before this make sure you have created 2-3 files in HDFS so that you can check and validate the file contain with normal file w.r.t size 

9. Copying data from one node to another node in HDFS 

Syntax:

- hdfs dfs – distcp <node1 file path details > <node 2 file path details > 

10. Display the contain of the data 

Syntax: 

- hdfs dfs – cat < File path details with file name > 

Example:

- hdfs  dfs –cat /user/root/hadoop_mahendhar/ First_hadoop.txt 

Screen Shot:

11. Change the group, owner, permission of the file or directory 

Syntax: 

- hdfs dfs –chgrp [-R] <New group Name > <File or directory > 

- hdfs dfs –chmod [ -R] <Fileor directory> 

- hdfs dfs –chown <New Owner name> < File or directory> 

12. Copying and Moving files within HDFS 

Syntax:

- hdfs dfs –cp <First file path details > < Destination file details >

- hdfs dfs –mv <First file path details > < Destination file details > 

13. Empty the hadoop thrash 

Syntax: 

- hdfs dfs –expunge 

Screen Shot:

Apache Hadoop Terms/Abbreviations click here 

Apache Hadoop Abbreviations/Terms

Hadoop Terms

HDFS - Hadoop Distributed File System GFS - Google File System NN - NameNode DN - Data Node SNN - Secondary NameNode JT - Job Tracker TT - Task Tracker HA NN - Highly Available NameNode (or NN HA - NameNode Highly Available) REST - Representational State Transfer HiveQL - Hive SQL HAR - Hadoop Archive ORC - Optimized Row Columnar JSON - Java Script Object Notation CDH - Cloudera’s Distribution Including Apache Hadoop ZKFC - ZooKeeper Failover Controller FUSE - Filesystem In Userspace YARN - Yet Another Resource Negotiator Amazon EC2 - Amazon Elastic Compute Cloud Amazon S3 - Amazon Simple Storage Service WASB - Windows Azure Storage Blobs (WASB) EMR - Elastic MapReduce JAR - Java ARchive RPC - Remote Procedure Call UDFs - user-defined functions ETL - Extract/Transform/Load

Hadoop -1.0.4.tar.gz Directory Structure click here

Hadoop versions

This article will help you understand what are all Apache HADOOP versions.

Hadoop 2.7.2 (released on 25 January, 2016) 2.X.X -  current stable version  Hadoop 2.7.1 (released on 06 July, 2015)  Hadoop 2.7.0 (released on 21 April 2015)  Hadoop 2.6.4 (released on 11 February, 2016)  Hadoop 2.6.3 (released on 17 December, 2015)  Hadoop 2.6.2 (released on 28 October, 2015)  Hadoop 2.6.1 (released on 23 September, 2015)  Hadoop 2.6.0 (released on 18 November, 2014)  Hadoop 2.5.2 (released on 19 November, 2014)  Hadoop 2.5.1 (released on 12 September, 2014)  Hadoop 2.5.0 (released on 11 August, 2014)  Hadoop 2.4.1 (released on 30 June, 2014)  Hadoop 2.4.0 (released on 07 April, 2014)  Hadoop 2.3.0 (released on 20 February, 2014)  Hadoop 2.2.0 (released on 15 October, 2013)  Hadoop 2.1.1 (released on 23 September, 2013) 2.X.X - beta version  Hadoop 2.1.0 (released on 25 August, 2013)  Hadoop 2.0.6 (released on 23 August, 2013) 2.X.X -  alpha version  Hadoop 2.0.5 (released on 6 June, 2013)  Hadoop 2.0.4 (released on 25 April, 2013)  Hadoop 2.0.3-alpha (released on 14 February, 2013)          Hadoop 2.0.2-alpha (released on 9 October, 2012)  Hadoop 2.0.1-alpha (released on 26 July, 2012)  Hadoop 2.0.0-alpha (released on 23 May, 2012)  Hadoop 1.2.1 (released on 1 Aug, 2013)                         1.2.1 - Stable version  Hadoop 1.2.0 (released on 13 May, 2013)  Hadoop 1.1.2 (released on 15 February, 2013)                1.1.X -  beta version  Hadoop 1.1.1 (released on 1 December, 2012)  Hadoop 1.1.0 (released on 13 October, 2012)  Hadoop 1.0.4 (released on 12 October, 2012)                 1.0.X -  stable version  Hadoop 1.0.3 (released on 16 May, 2012)  Hadoop 1.0.2 (released on 3 Apr, 2012)  Hadoop 1.0.1 (released on 10 Mar, 2012)  Hadoop 1.0.0 (released on 27 December, 2011)  Hadoop 0.23.11(released on 27 June, 2014)  Hadoop 0.23.10(released on  11 December, 2013)  Hadoop 0.23.9 (released on 8 July, 2013)  Hadoop 0.23.8 (released on 5 June, 2013)  Hadoop 0.23.7 (released on  18 April, 2013)  Hadoop 0.23.6 (released on 7 February, 2013)           0.23.X - simmilar to 2.X.X but missing NN HA  Hadoop 0.23.5 (released on 28 November, 2012)  Hadoop 0.23.4 (released on 15 October, 2012)  Hadoop 0.23.3 (released on 17 September, 2012)  Hadoop 0.23.1 (released on 27 Feb, 2012)  Hadoop 0.22.0 (released on 10 December, 2011)        0.22.X - does not include security  Hadoop 0.23.0 (released on 11 Nov, 2011)  Hadoop 0.20.205.0 (released on 17 Oct, 2011)  Hadoop 0.20.204.0 (released on 5 Sep, 2011)  Hadoop 0.20.203.0 (released on 11 May, 2011)          0.20.203.X - old legacy stable version  Hadoop 0.21.0 (released on 23 August, 2010)  Hadoop 0.20.2 (released on 26 February, 2010)         0.20.X - old legacy version  Hadoop 0.20.1 (released on 14 September, 2009)  Hadoop 0.19.2 (released on 23 July, 2009)  Hadoop 0.20.0 (released on 22 April, 2009)  Hadoop 0.19.1 (released on 24 February, 2009)  Hadoop 0.18.3 (released on 29 January, 2009)  Hadoop 0.19.0 (released on 21 November, 2008)  Hadoop 0.18.2 (released on 3 November, 2008)  Hadoop 0.18.1 (released on 17 September, 2008)  Hadoop 0.18.0 (released on 22 August, 2008)  Hadoop 0.17.2 (released on 19 August, 2008)  Hadoop 0.17.1 (released on 23 June, 2008)  Hadoop 0.17.0 (released on 20 May, 2008)  Hadoop 0.16.4 (released on 5 May, 2008)  Hadoop 0.16.3 (released on 16 April, 2008)  Hadoop 0.16.2 (released on 2 April, 2008)  Hadoop 0.16.1 (released on 13 March, 2008)  Hadoop 0.16.0 (released on 7 February, 2008)  Hadoop 0.15.3 (released on 18 January, 2008)  Hadoop 0.15.2 (released on 2 January, 2008)  Hadoop 0.15.1 (released on 27 November, 2007)  Hadoop 0.14.4 (released on 26 November, 2007)  Hadoop 0.15.0 (released on 29 October 2007)  Hadoop 0.14.3 (released on 19 October, 2007)  Hadoop 0.14.1 (released on 4 September, 2007)

Data Types in Hadoop click here
Hadoop -1.0.4.tar.gz Directory Structure click here

Linux File permissions

What I'll cover in this article is how to identify permissions for files & directories and how to change them, as well as changing ownerships, groups, etc. Depending on what you want to do, you'll want to make sure you have the appropriate permissions (obviously), so let's find out how to change them.

1.     File Permission Groups

2.     File Permission Types

3.     Viewing File Permission

4.     Modifying File Permission

1.   File Permission Groups:

Each file and directory has three user based permission groups:

• owner - The Owner permissions apply only the owner of the file or directory.
• group - The Group permissions apply only to the group that has been assigned to the file or directory.
• all users - The All Users permissions apply to all other users on the system.

Example:

-rwxrw-r-- file1.txt

In the whole we have 10 digits ,

( 1 )First field ( - ) gives the type of file ,

( 2 - 4 ) Next three field tells about Owner’s access on the file .

( 5 - 7 ) Next three fields tells about access of groups on the file .

( 8 - 10 ) Last three fields tells about all other users access on the file . In the above example,

Owner has complete access ie. Read, Write and Execute permission. 

Group has read and write permission on the file file1.txt .

All other users have read-only access on the file .

2. File Permission Types:

Each file or directory has three basic permission types:

•  read - The Read permission refers to a user's capability to read the contents of the file.
• write - The Write permissions refer to a user's capability to write or modify a file or directory.
• execute - The Execute permission affects a user's capability to execute a file or view the contents of a directory.

Example :

rwxrwxrwx file1.txt 

In the above ,

r – Stands for Read access. 

w – Stands for Write access.

x – Stands for Execute permission.

3.   Viewing File Permission:

ls command helps to list the files in the directory . It also helps to display the files with its permission. 

COMMAND:

ls –l helps to list the files with their permissions .

    Example:

             A directory named sample under root directory contains list of below files. 

sample1.txt

sample2.txt 

sample3.txt

Linux $ ls –l /sample

rwxrwxrwx sample1.txt 

rwxrw-r-- sample2.txt 

rwx------ sample3.txt

In sample1.txt , Every users has complete access on it .

In sample2.txt , Owner has complete access , Group has read write access , all others has read only access .

In sample3.txt , Only owner of the file has permission to access the file .

4.   Modifying File Permission:

In Linux, File permission can be modified using command chmod. In Linux, chmod stands for change mode.

There are two types of method to change file permission using chmod. They are,

        4.1    Using Assignment operator,

        4.2    Using Numeric values.

4.1   chmod - Using Assignment operator:

        Assignment operators used in modifying file permission using chmod are “+” and “-”.

        “+” – This operator helps to provide permission to the file or directory. 

         “-” – This operator helps to remove permission to the file or directory. 

          The Permission Groups used here are:

          u - Owner 

          g - Group

          o or a - All Users

The Permission Types that are used here are: 

        r - Read

       w - Write

       x – Execute

Example:

Using above permission groups and types in chmod we change the permission of the file.

Linux $ ls –l /sample 

       rwxrwxrwx sample1.txt

       rwxrw-r-- sample2.txt 

       rwx------ sample3.txt

       Providing file permission using + operator, 

       Linux $ chmod a+rw sample3.txt

      The above command provides read and write permission to global users.

Linux $ ls –l sample3.txt 

rwx---rw- sample3.txt

Removing file permission using - operator,

Linux $ ls –l /sample

rwxrwxrwx sample1.txt

rwxrw-r-- sample2.txt 

rwx---rw- sample3.txt

Removing file permission using - operator, Linux $ chmod a-wx sample1.txt

The above command removes read and write access to global users.

Linux $ ls –l sample1.txt 

rwxrwxr-- sample1.txt

4.2   chmod - Using Numeric values:

Numeric values for type of permissions are, 

r = 4

w = 2

x = 1

Below are the numeric values for different type of permissions, 

7 = 4+2+1 (read/write/execute)

6 = 4+2 (read/write)

5 = 4+1 (read/execute)

4 = 4 (read)

3 = 2+1 (write/execute)

2 = 2 (write)

1 = 1 (execute)

Example:

chmod 400 sample.txt – read by owner

chmod 040 sample.txt – read by group

chmod 004 sample.txt – read by anybody (other) 

chmod 200 sample.txt – write by owner

chmod 020 sample.txt – write by group 

chmod 002 sample.txt – write by anybody 

chmod 100 sample.txt – execute by owner

chmod 030 sample.txt – write and execute by group

chmod 001 sample.txt – execute by anybody 

chmod 666 sample.txt – read and write by anybody

chmod 755 sample.txt – rwx for owner, rx for group and rx for the world 

chmod 777 sample.txt – read, write, execute for all.