AES – MR: A Novel Encryption Scheme for securing Data in HDFS Environment using MapReduce

Abstract

Data security is an important issue as far as storage of sensitive data is concerned. Hadoop is usually utilized for storage, large amount of data using its storage technology, namely Hadoop Distributed file System HDFS. Hadoop by default does not contain any security mechanism but as it has grown very much and it is the first choice of the business analyst and industries to store and manage data it is necessary to introduce security solutions to Hadoop in order to secure the important data in the Hadoop environment. Authentication, Authorization, Data Encryption, Security against various attacks are the key levels of data and information security in the Hadoop environment. Efforts have been made in order to attain each and every level of security over the period of recent years. Kerberos is one such effort in order to attain Authentication and Authorization and it succeeded in doing so, but with the attackers having new technologies and hacking tools attackers can easily bypass the security provided by Hadoop's Kerberos Authentication system and then the data at storage level is unencrypted can easily be stolen or damaged which is a big concern.Encryption of large data stored in HDFS is actually a process which takes a lot of time and this time consuming nature of encryption should be controlled by encrypting the data using a parallel method. This study discusses a new technique to perform encryption in parallel using AES-MR (an Advanced Encryption standard based encryption using MapReduce) technique in MapReduce paradigm..The time taken for Performing the encryption and decryption process is relatively less for user generated content. Results show that AES-MR encryption process is found to be faster with mapper function alone in comparison with running the encryption process under mapper function and reducer function. Here a new encryption scheme is given by the combination of AES and MapReduce in order to secure data in HDFS environment.The results generated for encryption on different data proves that the proposed technique is well suited for protecting user generated sensitive data deployed in the HDFS environment.