Estimation of Secure Data Deduplication in Big Data

Abstract

Bigdata is linked with the entireties of composite data sets. In bigdata environment, data is in the form of unstructured data and may contain number of duplicate copies of same data. To manage such a complex unstructured data hadoop is to be used. A hadoop is an open source platform specially designed for bigdata environment. Hadoop can handle unstructured data very efficiently as compare to tradition data processing tools. To reduce duplicity of data concept of deduplication is used. In this paper an evaluation of different chunking and deduplication techniques has been presented.