Byte-index Chunking Algorithm for Data Deduplication System

Lkhagvasuren, Ider; So, Jung Min; Lee, Jeong Gun; Ko, Young Woong

doi:10.14257/ijsia.2013.7.5.38

Cited by 9 publications

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The previously proposed chunking methods like BSW [8], TTTD [9], TTTD-S [10], Byte index chunking [11], FastCDC [12] etc. are used to calculate the size of chunks.…”

Section: Proposed Workmentioning

confidence: 99%

See 1 more Smart Citation

A Novel approach of data deduplication for distributed storage

Singhal¹,

Kaushik²,

Sharma³

2018

IJET

View full text Add to dashboard Cite

Due to drastic growth of digital data, data deduplication has become a standard component of modern backup systems. It reduces data redundancy, saves storage space, and simplifies the management of data chunks. This process is performed in three steps: chunking, fingerprinting, and indexing of fingerprints. In chunking, data files are divided into the chunks and the chunk boundary is decided by the value of the divisor. For each chunk, a unique identifying value is computed using a hash signature (i.e. MD-5, SHA-1, SHA-256), known as fingerprint. At last, these fingerprints are stored in the index to detect redundant chunks means chunks having the same fingerprint values. In chunking, the chunk size is an important factor that should be optimal for better performance of deduplication system. Genetic algorithm (GA) is gaining much popularity and can be applied to find the best value of the divisor. Secondly, indexing also enhances the performance of the system by reducing the search time. Binary search tree (BST) based indexing has the time complexity of (log ) which is minimum among the searching algorithm. A new model is proposed by associating GA to find the value of the divisor. It is the first attempt when GA is applied in the field of data deduplication. The second improvement in the proposed system is that BST index tree is applied to index the fingerprints. The performance of the proposed system is evaluated on VMDK, Linux, and Quanto datasets and a good improvement is achieved in deduplication ratio.

show abstract

“…The previously proposed chunking methods like BSW [8], TTTD [9], TTTD-S [10], Byte index chunking [11], FastCDC [12] etc. are used to calculate the size of chunks.…”

Section: Proposed Workmentioning

confidence: 99%

“…Deduplication detection ratio of variable-size chunks is better than fixed-size chunks. Various methods like BSW [8], TTTD [9], TTTD-S [10], Byteindex [11], FastCDC [12] have been proposed to decide the chunk boundary. The size of chunks is generally decided by divisor 'D'.…”

Section: Introductionmentioning

confidence: 99%

A Novel approach of data deduplication for distributed storage

Singhal¹,

Kaushik²,

Sharma³

2018

IJET

View full text Add to dashboard Cite

show abstract

“…Weny et al [15] developed an efficient secure deduplicationsscheme for big data outsourcing in the cloud computing. Byte-Index Chunking [18] Byte-Index is a technique which first determines identical blocks and then performs de-duplication on identical/unique block.…”

Section: Related Workmentioning

confidence: 99%

Estimation of Secure Data Deduplication in Big Data

Kumar¹,

Shalini²

2017

IJARCSSE

View full text Add to dashboard Cite

Bigdata is linked with the entireties of composite data sets. In bigdata environment, data is in the form of unstructured data and may contain number of duplicate copies of same data. To manage such a complex unstructured data hadoop is to be used. A hadoop is an open source platform specially designed for bigdata environment. Hadoop can handle unstructured data very efficiently as compare to tradition data processing tools. To reduce duplicity of data concept of deduplication is used. In this paper an evaluation of different chunking and deduplication techniques has been presented.

show abstract