NV-Dedup: High-Performance Inline Deduplication for Non-Volatile Memory

Wang, Chundong; Wei, Qingsong; Yang, Jun; Chen, Cheng; Yang, Yechao; Xue, Minhui

doi:10.1109/tc.2017.2774270

Cited by 17 publications

(5 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Systems for Comparison. To compare ICID with chunkbased deduplication schemes, we design a general framework to track data to be written and split it into chunks which are deduplicated using a similar approach like NV-Dedup [20]. We use XXHash [35] to calculate fingerprints of these chunks for high throughput.…”

Section: Methodsmentioning

confidence: 99%

“…Moreover, ICID also cannot be used for scenarios such as compression/decompression, encryption/decryption, and pre-existing redundancy in a file. In these cases, chunk-based data deduplication approaches [9], [18], [19], [20] are more appropriate.…”

Section: Data Redundancy Characteristicsmentioning

confidence: 99%

“…Dmdedup [26] is implemented in the block layer with fixedsize chunking. NV-Dedup [20] achieves deduplication for an NVM-based filesystem. FPC [27] integrates a deduplication scheme into EXT4/F2FS filesystems to reduce the write traffic.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

I/O Causality Based In-Line Data Deduplication for Non-Volatile Memory Enabled Storage Systems

Liu,

Jin,

et al. 2024

IEEE Trans. Comput.

View full text Add to dashboard Cite

Data deduplication technologies are widely exploited to reduce capacity demands for storage. Previous chunk-based offline deduplication technologies often cause serious performance overhead due to data chunking and indexing. Particularly, they are not efficient for non-volatile memory (NVM) based storage systems because they cannot fully exploit the byte-addressability feature of NVMs for fine-grained deduplication. In this paper, we propose I/O Causality based In-line Deduplication (ICID) to maximize the deduplication ratio for NVM-based storage systems. Unlike previous inline deduplication schemes that use hash indexes to identify duplicate data slices, ICID records memory-copy operations in a B-tree structure to achieve causality-based inline deduplication. We propose two novel techniques to manage memory-copy records in the B-tree efficiently. First, to speed up the B-tree lookup, we group memory-copy records targeted to the same page in a B-tree node to improve data locality. Second, we exploit the spatial locality of memory accesses to identify outdated memory-copy records, and delete them in time to reduce memory consumption of the B-tree. We evaluate ICID in a system equipped with Intel Optane DC Persistent Memory Modules. For a typical KV store-LevelDB, our experimental results show that ICID achieves up to 16× higher deduplication ratio and reduces the time cost of data deduplication by 47% on average compared with state-of-the-art deduplication schemes.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Data Redundancy Characteristicsmentioning

confidence: 99%

See 1 more Smart Citation

I/O Causality Based In-Line Data Deduplication for Non-Volatile Memory Enabled Storage Systems

Liu,

Jin,

et al. 2024

IEEE Trans. Comput.

View full text Add to dashboard Cite

show abstract

“…Different from the general postprocess method, inline deduplication is performed before writing the data chunk into storage space, 12,24 therefore, applying it to PM system not only improves storage space utilization but also extends the device life by reducing the writing number. One downside is that performing inline deduplication affects the I/O latency.…”

Section: Background and Motivationmentioning

confidence: 99%

“…NV‐Dedup 24 aimed to design a deduplication system for PM. It provided fine metadata management and other low‐overhead workloads.…”

Section: Related Workmentioning

confidence: 99%

Low‐overhead inline deduplication for persistent memory

Chen

et al. 2020

Trans Emerging Tel Tech

View full text Add to dashboard Cite

Persistent memory (PM) can play an important role in storing the edge data of IoT devices, but their capacities are restrained by the embedded system, which only provides the limited host space and energy resource. Therefore, a deduplication system is required to improve the PM utilization and energy consumption. According to our observations, the current inline deduplication algorithms incur nonnegligible latency when applied to the PM file systems (PMFSs), mainly because (i) the low tolerance to the extra deduplication I/O path, (ii) the long fingerprint calculation, and (iii) the unnecessary duplication detection when writing content with low duplication ratio. To improve this issue, we propose LO‐Dedup, a novel and low‐overhead deduplication system for PM. LO‐Dedup adopts PM‐friendly design, both in fine‐grained data load/store and fast hash scheme to reduce overhead. Moreover, LO‐Dedup can adaptively sample duplication detection according to the recent contents. We implement a prototype of LO‐Dedup in PMFS, which is a well‐known PMFS. The experimental results show that the write performance only has a slight drop when writing data and saves up to 45% space in PMFS.

show abstract

Towards Optimizing Deduplication on Persistent Memory

Wang

et al. 2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

NV-Dedup: High-Performance Inline Deduplication for Non-Volatile Memory

Cited by 17 publications

References 24 publications

I/O Causality Based In-Line Data Deduplication for Non-Volatile Memory Enabled Storage Systems

I/O Causality Based In-Line Data Deduplication for Non-Volatile Memory Enabled Storage Systems

Low‐overhead inline deduplication for persistent memory

Towards Optimizing Deduplication on Persistent Memory

Contact Info

Product

Resources

About