How to Change the priority of Hadoop Jobs (Hive,MapReduce,pig)?

Hadoop provides a command line utility for changing the priority of running job. There are total 5 different priority level are there. These priority levels are listed below:

VERY_HIGH
HIGH
NORMAL
LOW
VERY_LOW

We can change the priority of the running hadoop job syntax is shown below:

 hadoop job -set-priority <job-id> <priority>  

Example:

 hadoop job -set-priority job_20170111540_64444 VERY_HIGH  

From the names of the priority levels, it is quite obvious that highest priority is given to the jobs whose priority level is VERY_HIGH and least priority is given to the jobs whose priority level is VERY_LOW.

Priority in Map Reduce Program:

In order to set priority for a job on Hadoop 1 cluster, you can use the following example:

 -> SET mapred.job.priority=<priority_value>;  

In order to set priority for a job on Hadoop 2 cluster, you can use the following example:

 -> SET mapreduce.job.priority=<priority_value>;  

Priority in Pig Program:

The below property is used to set the job priority is Pig Programming.

 grunt> SET job.priority 'high'  

The following <priority_value> values are supported:
very_low, low, normal, high, very_high

Priority for Hive Query:

In order to set priority for a HIVE query on Hadoop 1 cluster, you can use the following example:

 hive> SET mapred.job.priority=VERY_HIGH;  

In order to set priority for a HIVE query on Hadoop 2 cluster, you can use the following example:

 hive> SET mapreduce.job.priority=VERY_HIGH;  

If you are facing any problems in changing the priority of the hadoop jobs, then please comment here.

DISTRIBUTED CACHE IN HADOOP

Introduction :

In the Using third part jars and files in your MapReduce application(Distributed cache) entry i blogged about how to use Distributed Cache in Hadoop using DistributedCache API . But you can also have option of using command line option. You will have to use following steps to use DistributedCache programmatically In order to use it, first change your MapReduce Driver class to add job.addCacheFile()

Hadoop Map Reduce Project provides us this facility with something called as DistributedCache.
This Distributed Cache is configured with Job Configuration, What it does is, it provides read only data to all machine on the cluster.

Step 1 : Put file to HDFS

In order to use a file with DistributedCache API, it has to available on either hdfs:// or http:// URL, that is accessible to all the cluster members. So first step was to upload the file that you are interested in into HDFS, in my case i used following command to copy the /tmp/file1 file to hdfs.

 # hdfs dfs -put /tmp/file1 /cachefile1  

Step 2: Add cachefile in Job Configuration

Next step is to change the Driver class and add job.addCacheFile(new URI("/cachefile1")); call, this call takes the hdfs url of the file that you just uploaded to HDFS and passes it to DistributedCache class.

 Configuration conf = new Configuration();  
 Job job = new Job(conf, "wordcount");  
 DistributedCache.addCacheFile(new URI("/cachefile1"),job.getConfiguration());  

Step 3: Access Cached file

Now in your Mapper class you can read the file1 using normal File API

 Path[] cacheFiles = context.getLocalCacheFiles();  
 FileInputStream fileStream = new FileInputStream(cacheFiles[0].toString());  

You may like reading HADOOP directory structure Hadoop -1.0.4.tar.gz Directory Structure

If you know anyone who has started learning Hadoop and Java, why not help them out! Just share this post with them. Thanks for studying today!

HDFS Commands Reference

In this article, the basic syntax of hadoop file system i.e HDFS has been explained with examples and screen shot. This is very useful for the beginners who are interested to explore the big data world and HDFS is the gate to that world.

Hadoop is open source software [It is a java frame work] which runs on a cluster of commodity hardware machines. It provides both storage [HDFS] and processing [MAP REDUCE] in distributed manner. It has capable of processing huge volume of data that is ranging from Giga bytes to Peta bytes.

HDFS Commands

hdfs dfs

hadood fs:

hdfs dfs/hadoop fs

1.  Creating a directory [HDFS]

Syntax:

- hdfs dfs –mkdir <Directory name along with path details >

Example:

- hdfs dfs –mkdir /user/root/hadoop_mahendhar

Screenshot

2.  Listing the contains of the hadoop directory

Syntax:

- hdfs dfs –ls < argument like absolute path of the file >

Example:

- hdfs  dfs –ls /user/root/hadoop_mahendhar

Screen Shot:

3.  Create a file in local file system and put the file in HDFS

Create a file in local system by  vi <file_name>, add some texts and save, exit.

Syntax: vi First_hadoop.txt

Putting the normal file to hadoop file system

Syntax:

- hdfs dfs –put <Local file path with file name > <hadoop destination path with file name >

Example:

- hdfs dfs –put  First_hadoop.txt  /user/root/hadoop_mahendhar

Screen Shot:

4. Moving a normal file to hadoop file system 

Syntax: 

- hdfs dfs – moveFromLocal <Local file path with file name > <hadoop destination path with file name > 

Example:

- hdfs dfs  –moveFromLocal  /root/Second_hadoop.txt  /user/root/hadoop_mahendhar

Screenshot:

Note:

1. Before executing the above command, ensure that the second_hadoop.txt file is created in the Local normal file system. 

2. This operation will move the local file, so there is no local copy of the file exist after this operation. 

5. For listing all directories and sub-directories recursively 

Syntax:

- hdfs dfs –lsr <hadoop directory>

Example:

- hdfs  dfs -lsr /user/root/hadoop_mahendhar/ 

- Note: Create more directory and sub directory to validate this command correctly. 

6. Check the size of the file in HDFS 

Syntax: 

- hdfs dfs – du <File path with file name > 

Example:

- hdfs  dfs – du /user/root/hadoop_mahendhar/  

Screen Shot:

7. Download a file from HDFS to normal file system 

Syntax: 

- hdfs dfs – get <hadoop file path details with file name > < local file path details with file name> 

Example:

- hdfs dfs – get /user/root/hadoop_mahendhar/Second_hadoop.txt /root/local_files/ 

Screen Shot:

8. Getting a directory of files from HDFS and merge into a single file in normal file system 

Syntax: 

- hdfs dfs – getmerge <HDFS file directory > < Local file path with file details > < add new line> 

Example:

- hdfs dfs – getmerge /user/root/hadoop_mahendhar/ /root/local_files/hadoop_merge_file.txt 

Screen Shot:

Note:

1. The add newline is optional and it will just add a new line at the end of the each file. 

2. Before this make sure you have created 2-3 files in HDFS so that you can check and validate the file contain with normal file w.r.t size 

9. Copying data from one node to another node in HDFS 

Syntax:

- hdfs dfs – distcp <node1 file path details > <node 2 file path details > 

10. Display the contain of the data 

Syntax: 

- hdfs dfs – cat < File path details with file name > 

Example:

- hdfs  dfs –cat /user/root/hadoop_mahendhar/ First_hadoop.txt 

Screen Shot:

11. Change the group, owner, permission of the file or directory 

Syntax: 

- hdfs dfs –chgrp [-R] <New group Name > <File or directory > 

- hdfs dfs –chmod [ -R] <Fileor directory> 

- hdfs dfs –chown <New Owner name> < File or directory> 

12. Copying and Moving files within HDFS 

Syntax:

- hdfs dfs –cp <First file path details > < Destination file details >

- hdfs dfs –mv <First file path details > < Destination file details > 

13. Empty the hadoop thrash 

Syntax: 

- hdfs dfs –expunge 

Screen Shot:

Apache Hadoop Terms/Abbreviations click here