DISTRIBUTED CACHE IN HADOOP

Introduction :

In the Using third part jars and files in your MapReduce application(Distributed cache) entry i blogged about how to use Distributed Cache in Hadoop using DistributedCache API . But you can also have option of using command line option. You will have to use following steps to use DistributedCache programmatically In order to use it, first change your MapReduce Driver class to add job.addCacheFile()

Hadoop Map Reduce Project provides us this facility with something called as DistributedCache.
This Distributed Cache is configured with Job Configuration, What it does is, it provides read only data to all machine on the cluster.

Step 1 : Put file to HDFS

In order to use a file with DistributedCache API, it has to available on either hdfs:// or http:// URL, that is accessible to all the cluster members. So first step was to upload the file that you are interested in into HDFS, in my case i used following command to copy the /tmp/file1 file to hdfs.

 # hdfs dfs -put /tmp/file1 /cachefile1  

Step 2: Add cachefile in Job Configuration

Next step is to change the Driver class and add job.addCacheFile(new URI("/cachefile1")); call, this call takes the hdfs url of the file that you just uploaded to HDFS and passes it to DistributedCache class.

 Configuration conf = new Configuration();  
 Job job = new Job(conf, "wordcount");  
 DistributedCache.addCacheFile(new URI("/cachefile1"),job.getConfiguration());  

Step 3: Access Cached file

Now in your Mapper class you can read the file1 using normal File API

 Path[] cacheFiles = context.getLocalCacheFiles();  
 FileInputStream fileStream = new FileInputStream(cacheFiles[0].toString());  

You may like reading HADOOP directory structure Hadoop -1.0.4.tar.gz Directory Structure

If you know anyone who has started learning Hadoop and Java, why not help them out! Just share this post with them. Thanks for studying today!

How to drop all tables in hive at a time in a linux environment ?

Use the following to delete all the tables in a linux environment.

hive -e 'show tables' | xargs -I '{}' hive -e 'drop table {}'

How to add a new column to a table in Hive

Use the following command to add a new column to a table in Hive

    ALTER TABLE students ADD COLUMNS (new_col INT);

How to copy a table in Hive to another table or create temp table?

If you want to store results of a query in a table in Hive then

    Create a schema of the temp table using command CREATE TABLE ..
    Execute the following command

INSERT OVERWRITE TABLE temp_tablename SELECT * FROM table_name limit 10

How to create output in gzip files in Hadoop Hive ?

How to create output in gzip files in Hadoop Hive

Sometimes its required to output hive results in gzip files  to reduce the file size so that the files can be transferred over network.
To do this, run the following commands in hive before running the query. The following code sets these options and then runs the hive query. The output of this hive query will be stored in gzip files.

set mapred.output.compress=true;
set hive.exec.compress.output=true;
set mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec;
set io.compression.codecs=org.apache.hadoop.io.compress.GzipCodec;
INSERT OVERWRITE DIRECTORY 'hive_out' select * from tables limit 10000;

Java Program to List Contents of Directory in Hadoop (HDFS)