Optimization of De-duplication Technology Based on CDC Blocking Algorithm

Nie, Junyang; Wu, Lijuan; Jing-zhang, Liang

doi:10.1109/cisp-bmei48845.2019.8965818

Cited by 3 publications

(2 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…33 Instead of comparing bytes by bytes and transparently removing redundant objects, they often compare computed fingerprints to find redundant things. 34,35 Researchers primarily concentrate on enhancing data chunking methods to raise the deduplication ratio. According to Zhou et al, 36 the chunking stage of most modern chunking algorithms uses the Rabin method sliding window, which uses a lot of CPU resources and affects performance.…”

Section: Data Deduplicationmentioning

confidence: 99%

See 1 more Smart Citation

Blockchain‐based data deduplication using novel content‐defined chunking algorithm in cloud environment

Prakash J,

et al. 2023

Int J Network Mgmt

View full text Add to dashboard Cite

The cloud environment is inherently dynamic as users are added immensely in a short duration. It is indeed difficult to manage such user profiles and associated data. Meanwhile, the cloud data expand at a twofold‐to‐threefold rate on average, making storage space management and data integrity maintenance a mandatory task but also risky. The main approaches for addressing these data maintenance challenges in a cloud context are deduplication and data protection. In order to manage storage space, finding and removing identical copies of the same data from the cloud are possible, resulting in a reduction in the amount of storage space needed. Furthermore, duplicate copies are considerably reduced in cloud storage owing to data deduplication. Here, a decentralized ledger public blockchain network is introduced to protect the Integrity of data stored in cloud storage. This research proposes data deduplication using speedy content‐defined Chunking (SpeedyCDC) algorithm in the public blockchain. Many people and businesses outsource sensitive data to remote cloud servers because it considerably eliminates the hassle of managing software and infrastructure. However, the ownership and control rights of users data are nonetheless divided because it is outsourced to cloud storage providers (CSPs) and kept on a distant cloud. As a result, users have a great deal of difficulty in verifying the Integrity of sensitive data. Analysis using datasets from Geospatial Information Systems (GIS) revealed that the throughput increased by 5%–6% over that of the fastCDC technique, which offered Integrity since a blockchain network secured it.

show abstract

Section: Data Deduplicationmentioning

confidence: 99%

“…On the other hand, it is highly effective and inexpensive 33 . Instead of comparing bytes by bytes and transparently removing redundant objects, they often compare computed fingerprints to find redundant things 34,35 . Researchers primarily concentrate on enhancing data chunking methods to raise the deduplication ratio.…”

Section: Related Workmentioning

confidence: 99%

Blockchain‐based data deduplication using novel content‐defined chunking algorithm in cloud environment

Prakash J,

et al. 2023

Int J Network Mgmt

View full text Add to dashboard Cite

show abstract

Storage Enhancement in the Cloud Using Machine Learning Technique and Novel Hash Algorithm for Cloud Data Security

Hema

Márquez

2022

Proceedings of the Sixteenth International Conference on Management Science and Engineering Management – Volume 1

View full text Add to dashboard Cite

Data Deduplication System Based on Content-Defined Chunking Using Bytes Pair Frequency Occurrence

Saeed

George

2020

Symmetry

View full text Add to dashboard Cite

Every second, millions of data are being generated due to the use of emerging technologies. It is very challenging to store and handle such a large amount of data. Data deduplication is a solution for this problem. It is a new technique that eliminates duplicate data and stores only a single copy of data, reducing storage utilization and the cost of maintaining redundant data. Content-defined chunking (CDC) has been playing an important role in data deduplication systems due to its ability to detect high redundancy. In this paper, we focused on deduplication system optimization by tuning relevant factors in CDC to identify chunk cut-points and introduce an efficient fingerprint using a new hash function. We proposed a novel bytes frequency-based chunking (BFBC) algorithm and a new low-cost hashing function. To evaluate the efficiency of the proposed system, extensive experiments were done using two different datasets. In all experiments, the proposed system persistently outperformed the common CDC algorithms, achieving a better storage gain ratio and enhancing both chunking and hashing throughput. Practically, our experiments show that BFBC is 10 times faster than basic sliding window (BSW) and approximately three times faster than two thresholds two divisors (TTTD). The proposed triple hash function algorithm is five times faster than SHA1 and MD5 and achieves a better deduplication elimination ratio (DER) than other CDC algorithms. The symmetry of our work is based on the balance between the proposed system performance parameters and its reflection on the system efficiency compared to other deduplication systems.

show abstract

Optimization of De-duplication Technology Based on CDC Blocking Algorithm

Cited by 3 publications

References 7 publications

Blockchain‐based data deduplication using novel content‐defined chunking algorithm in cloud environment

Blockchain‐based data deduplication using novel content‐defined chunking algorithm in cloud environment

Storage Enhancement in the Cloud Using Machine Learning Technique and Novel Hash Algorithm for Cloud Data Security

Data Deduplication System Based on Content-Defined Chunking Using Bytes Pair Frequency Occurrence

Contact Info

Product

Resources

About