DISTRIBUTED CACHE IN HADOOP

Introduction :

In the Using third part jars and files in your MapReduce application(Distributed cache) entry i blogged about how to use Distributed Cache in Hadoop using DistributedCache API . But you can also have option of using command line option. You will have to use following steps to use DistributedCache programmatically In order to use it, first change your MapReduce Driver class to add job.addCacheFile()

Hadoop Map Reduce Project provides us this facility with something called as DistributedCache.
This Distributed Cache is configured with Job Configuration, What it does is, it provides read only data to all machine on the cluster.

Step 1 : Put file to HDFS

In order to use a file with DistributedCache API, it has to available on either hdfs:// or http:// URL, that is accessible to all the cluster members. So first step was to upload the file that you are interested in into HDFS, in my case i used following command to copy the /tmp/file1 file to hdfs.

 # hdfs dfs -put /tmp/file1 /cachefile1  

Step 2: Add cachefile in Job Configuration

Next step is to change the Driver class and add job.addCacheFile(new URI("/cachefile1")); call, this call takes the hdfs url of the file that you just uploaded to HDFS and passes it to DistributedCache class.

 Configuration conf = new Configuration();  
 Job job = new Job(conf, "wordcount");  
 DistributedCache.addCacheFile(new URI("/cachefile1"),job.getConfiguration());  

Step 3: Access Cached file

Now in your Mapper class you can read the file1 using normal File API

 Path[] cacheFiles = context.getLocalCacheFiles();  
 FileInputStream fileStream = new FileInputStream(cacheFiles[0].toString());  

You may like reading HADOOP directory structure Hadoop -1.0.4.tar.gz Directory Structure

If you know anyone who has started learning Hadoop and Java, why not help them out! Just share this post with them. Thanks for studying today!

Encryption and Decryption in Java

Encryption and Decryption in Java

In this article I will be giving you basic idea of encryption and decryption in java and how it works.

Encryption is the process of transforming information using an algorithm to make it unreadable to anyone except those possessing special knowledge, usually referred to as a key,.The result of the process is encrypted information,process is called encryption. Reverse of it terms as Decryption.

The core class used for encryption and decryption is javax.crypto.Cipher.

A cipher is an object capable of carrying out encryption and decryption according to an encryption scheme (algorithm).

This class contains a representation of a cipher that uses a specific algorithm and operating mode.It includes methods to initialize and operate cipher.

Types of Encryption

Encryption can be symmetric or asymmetric,based on the user requirement.

Symmetric encryption: - Encryption and decryption can be done symmetrically,here the same key is used to encrypt and decrypt the data. Because both parties have the same key, the decryption essentially is performed by reversing some part of the encryption process. The DES(Data Encryption Standard) algorithm is an example of a symmetric key. It is supported by the Java Cryptography Extension (JCE).

Symmetric encryption is the oldest and best-known technique. A secret key, which can be a number, a word, or just a string of random letters, is applied to the text of a message to change the content in a particular way. This might be as simple as shifting each letter by a number of places in the alphabet. As long as both sender and recipient know the secret key, they can encrypt and decrypt all messages that use this key.

The encryption key is trivially related to the decryption key, in that they may be identical or there is a simple transform to go between the two keys. The keys, in practice, represent a shared secret between two or more parties that can be used to maintain a private information link.

Symmetric-key algorithms can be divided into stream cipher and block cipher. Stream ciphers encrypt the bytes of the message one at a time, and block ciphers take a number of bytes and encrypt them as a single unit. Blocks of 64 bits have been commonly used; the Advanced Encrypted Standard (AES) algorithm approved by NIST in December 2001 uses 128-bit blocks.

In the first case we are requesting a Cipher that implements the Data Encryption Standard (DES) algorithm and in the second line we want a Cipher that uses the Advanced Encryption Standard (AES) algorithm using the Electronic Code Book (ECB) method and the PKCS5 padding scheme. It is important to get to know the algorithms and operations modes available in the target device or emulator.

Algorithms: DES, 3DES/DESede, AES

Modes: ECB, CBC

Padding Schemes: No padding, PKCS5 padding.

Asymmetric encryption:-Here we use of asymmetric key algorithms instead of or in addition to symmetric key algorithms. Using the techniques of public key-private key cryptography. They do not require a secure initial exchange of one or more secret keys as is required when using in symmetric key algorithms.

The problem with secret keys is exchanging them over the Internet or a large network while preventing them from falling into the wrong hands. Anyone who knows the secret key can decrypt the message. One answer is asymmetric encryption, in which there are two related keys--a key pair. A public key is made freely available to anyone who might want to send you a message. A second, private key is kept secret, so that only you know it.

The distinguishing technique used in public key-private key cryptography is use of asymmetric key algorithms because the key used to encrypte a message is not the same as the key used to decrypte it. Each user has a pair of cryptographic keys— a public key and a private key. The private key is kept secret, while the public key may be widely distributed. Messages are encrypted with the recipient's public key and can only be decrypted with the corresponding private key. The keys are related mathematically, but the private key cannot be feasibly (i.e., in actual or projected practice) derived from the public key. It was the discovery of such algorithms which revolutinized the practice of cryptography beginning in the middle 1970s.

In contrast Symmetric encryption,variations of which have been used for some thousands of years, use a single secret key shared by sender and receiver (which must also be kept private, thus accounting for the ambiguity of the common terminology) for both encryption and decryption. To use a symmetric encryption scheme, the sender and receiver must securely share a key in advance.

An analogy to public-key encryption is that of a locked mailbox with a mail slot. The mail slot is exposed and accessible to the public; its location (the street address) is in essence the public key. Anyone knowing the street address can go to the door and drop a written message through the slot; however, only the person who possesses the key can open the mailbox and read the message.

Symmetric vs. asymmetric algorithms:

(1)  Unlike symmetric algorithms, asymmetric key algorithm uses a different key for encryption than for decryption. I.e., a user knowing the encryption key of an asymmetric algorithm can encrypt messages, but cannot derive the decryption key and cannot decrypt messages encrypted with that key.

(2)  Symmetric-key algorithms are generally much less computationally intensive than asymmetric key algorithms. In practice, asymmetric key algorithms are typically hundreds to thousands times slower than symmetric key algorithms.

A conventional encryption scheme has five major parts:

Plaintext - this is the text message to which an algorithm is applied.

Encryption Algorithm - it performs mathematical operations to conduct substitutions and transformations to the plaintext.

Secret Key - This is the input for the algorithm as the key dictates the encrypted outcome.

Ciphertext - This is the encrypted or scrambled message produced by applying the algorithm to the plaintext message using the secret key.

Decryption Algorithm - This is the encryption algorithm in reverse. It uses the ciphertext, and the secret key to derive the plaintext message.

Starting with encryption

 Step 1:- Create a Cipher instance from Cipher class, we have to specify the following information and separated by a slashes (/).

1.  Algorithm name

2.  Mode (optional)

3.  Padding scheme (optional)

  

You need to create a Cipher. You use a factory method of the Cipher class so that you can take advantage of different providers without changing the application. You create a Cipher like this:

Cipher cipher = Cipher.getInstance("DES");

If no mode or padding has been specified, provider-specific default values for the mode and padding scheme are used. For example, the SunJCE provider uses ECB as the default mode, and PKCS5Padding as the default padding scheme for DES, DES-EDE and Blowfish ciphers. This means that in the case of the SunJCE provider,

Cipher c1 = Cipher.getInstance("DES/ECB/PKCS5Padding");

DES = Data Encryption Standard

ECB = Electronic Codebook mode

PKCS5Padding = PKCS #5-style padding

 Step 2:- Create a DES Key.

You create a symmetric key, use a factory method from the KeyGenerator class and pass in the algorithm as a String. When you call the generateKey() method, you get back an object that implements the Key interface instead of the KeyPair interface. The call looks something like this:

SecretKey key =

KeyGenerator.getInstance ("DES").generateKey();

Step 3:- Convert String into Byte[] array format.

String input= “This is the private message";

byte [] textBytes =input.getBytes();

Private message input as string is encrypted with the key specified using the DES symmetric algorithm.

 Step 4:- Make Cipher in encrypt mode, and encrypt it with Cipher.doFinal() method.

Cipher is used to encrypt and decrypt data that is passed in as byte arrays. The two essential methods you must use are init (), to specify which operation will be called, and

  

doFinal (), to perform that operation. For example, the following two lines use the cipher and key instances you created to encrypt a byte array called textBytes. The result is stored in a byte array called encryptedBytes.

Suppose the string to encrypte is in input string object.

cipher.init(Cipher.ENCRYPT_MODE, key);
byte[] encryptedBytes =

cipher.doFinal( textBytes );

   

Decryption

First of all you need the same key as you used in the encryption,because DES is the

Symmetric encryption technique.

After encrypting the private message now we need to decrypte it, so

Make Cipher in decrypt mode, and decrypt it with Cipher.doFinal() method as well.

desCipher.init(Cipher.DECRYPT_MODE, key);

byte[] textDecrypted = desCipher.doFinal(textEncrypted);

System.out.println("Text Decryted : " + new String(textDecrypted));

The output string is the same as given in the input for encryption.

As “This is the private message” in the output appears.

Exception handling:-Also for catching the exceptions occurring during the encryption and decryption process.

You need to catch the following exceptions.

Ex:-NoSuchAlgorithmException, NoSuchPaddingException, InvalidKeyException,

IllegalBlockSizeException, BadPaddingException.

If you enjoyed this post, I’d be very grateful if you’d help it spread by emailing it to a friend, or sharing it on Twitter or Facebook. Thank you!