Design of Global Data Deduplication for a Scale-Out Distributed Storage System

Oh, Myoungwon; Park, Se Jin; Yoon, Jungyeon; Kim, Sangjae; Lee, Kang‐Won; Weil, Sage A.; Yeom, Heon Y.

doi:10.1109/icdcs.2018.00106

Cited by 17 publications

(8 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Data reliability. The absolute data deduplication may corrupt data reliability because it removes all extra copies appended by redundancy scheme [19]. The first challenge is the high data restore overhead with frequent I/Os, especially for hot data with great access frequencies [6].…”

Section: Discussionmentioning

confidence: 99%

“…We do not compare the works in literature [17]- [19], due to the low writing throughput in the inline deduplication. These works do not fit our requirements for edge storage, where the writing throughput should be a priority.…”

Section: A Experimental Settingsmentioning

confidence: 99%

“…EFdedup [44] estimates the probability distribution of the sources, thus balancing the network-storage cost of edge deduplication. Literature [19] controls the rate of background deduplication threads and conducts selective deduplication to minimize performance degradation. The above methods are limited in file access efficiency, since the split chunks are distributed evenly in the storage cluster.…”

Section: Related Workmentioning

confidence: 99%

“…The redundancies would be discarded directly in the write path when their fingerprints are detected. However, the big-volume fingerprint index may retard the writing throughput, because the lookup time inflates linearly with the number of stored fingerprints [19]. This situation is even worse at edge, where the fingerprint indexes can be scattered across edge servers.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

LOFS: A Lightweight Online File Storage Strategy for Effective Data Deduplication at Network Edge

Cheng

Guo

Xia

et al. 2022

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

Edge computing responds to users' requests with low latency by storing the relevant files at the network edge. Various data deduplication technologies are currently employed at edge to eliminate redundant data chunks for space saving. However, the lookup for the global huge-volume fingerprint indexes imposed by detecting redundancies can significantly degrade the data processing performance. Besides, we envision a novel file storage strategy that realizes the following rationales simultaneously: 1) space efficiency, 2) access efficiency, and 3) load balance, while the existing methods fail to achieve them at one shot. To this end, we report LOFS, a Lightweight Online File Storage strategy, which aims at eliminating redundancies through maximizing the probability of successful data deduplication, while realizing the three design rationales simultaneously. LOFS leverages a lightweight three-layer hash mapping scheme to solve this problem with constant-time complexity. To be specific, LOFS employs the Bloom filter to generate a sketch for each file, and thereafter feeds the sketches to the Locality Sensitivity hash (LSH) such that similar files are likely to be projected nearby in LSH tablespace. At last, LOFS assigns the files to real-world edge servers with the joint consideration of the LSH load distribution and the edge server capacity. Trace-driven experiments show that LOFS closely tracks the global deduplication ratio and generates a relatively low load std compared with the comparison methods.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: A Experimental Settingsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

LOFS: A Lightweight Online File Storage Strategy for Effective Data Deduplication at Network Edge

Cheng

Guo

Xia

et al. 2022

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

show abstract

“…Since cluster-wide deduplication has been adopted in the general storage community [7], [8], [9], [10], [11], [12], [13], [14], it is necessary to extend its advantages to the ML and DL storage architectures without affecting performance. Deduplication improves the storage capacity by removing duplicate data across the cluster.…”

Section: Introductionmentioning

confidence: 99%

Accelerating ML/DL Applications with Hierarchical Caching on Deduplication Storage Clusters

Hamandawana

Khan

Kim

et al. 2021

IEEE Trans. Big Data

View full text Add to dashboard Cite

Large scale machine learning (ML) and deep learning (DL) platforms face challenges when integrated with deduplication enabled storage clusters. In the quest to achieve smart and efficient storage utilization, removal of duplicate data introduces bottlenecks, since deduplication alters the I/O transaction layout of the storage system. Therefore, it is critical to address such deduplication overhead for acceleration of ML/DL computation in deduplication storage. Existing state of the art ML/DL storage solutions such as Alluxio and AutoCache adopt non deduplication-aware caching mechanisms, which lacks the much needed performance boost when adopted in deduplication enabled ML/DL clusters. In this paper, we introduce Redup, which eliminates the performance drop caused by enabling deduplication in ML/DL storage clusters. At the core, is a Redup Caching Manager (RDCM), composed of a 2-tier deduplication layoutaware caching mechanism. The RDCM provides an abstraction of the underlying deduplication storage layout to ML/DL applications and provisions a decoupled acceleration of object reconstruction during ML/DL read operations. Our Redup evaluation shows negligible performance drop in ML/DL training performances as compared to a cluster without deduplication, whilst significantly outperforming Alluxio and AutoCache in terms of various performance metrics.

show abstract