CMD

Chen, Licheng; Wei, Zhipeng; Cui, Zehan; Chen, Mingyu; Pan, Haiyang; Bao, Yungang

doi:10.1145/2576195.2576204

Cited by 26 publications

(1 citation statement)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The aforementioned technologies are widely used in enterprise storage device file servers, data blocks, NAS (RAID), and backup devices. Prior to HDFS, however, there was no special block state and network topology in Hadoop to ensure the reliability of backup instructions [21,22]. That is, the condition of deleting duplicate files is only to detect similar or identical data.…”

Section: Hdfs Hash Functionmentioning

confidence: 99%

Data Deduplication Technology for Cloud Storage

Bian

Shao

et al. 2020

Teh. vjesn.

View full text Add to dashboard Cite

With the explosive growth of information data, the data storage system has stepped into the cloud storage era. Although the core of the cloud storage system is distributed file system in solving the problem of mass data storage, a large number of duplicate data exist in all storage system. File systems are designed to control how files are stored and retrieved. Fewer studies focus on the cloud file system deduplication technologies at the application level, especially for the Hadoop distributed file system. In this paper, we design a file deduplication framework on Hadoop distributed file system for cloud application developer. Proposed RFD-HDFS and FD-HDFS two data deduplication solutions process data deduplication online, which improves storage space utilisation and reduces the redundancy. In the end of the paper, we test the disk utilisation and the file upload performance on RFD-HDFS and FD-HDFS, and compare HDFS with the disk utilisation of two system frameworks. The results show that the two-system framework not only implements data deduplication function but also effectively reduces the disk utilisation of duplicate files. So, the proposed framework can indeed reduce the storage space by eliminating redundant HDFS file.

show abstract

Section: Hdfs Hash Functionmentioning

confidence: 99%