Crocus: Enabling Computing Resource Orchestration for Inline Cluster-Wide Deduplication on Scalable Storage Systems

Hamandawana, Prince; Khan, Awais; Lee, Chang-Gyu; Park, Sungyoung; Kim, Young-Jae

doi:10.1109/tpds.2020.2972882

Cited by 10 publications

(4 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, some proposals exploit multiple-threading and pipelining technologies [15], [16], [18] to parallelize different deduplication stages, and thus improve the throughput of deduplication systems. A few works exploit heterogeneous computing resources [19], [20] such as GPUs/FPGAs to accelerate data deduplication. CROCUS [19] orchestrates CPUs and GPUs to enhance the throughput of deduplication systems.…”

Section: Related Workmentioning

confidence: 99%

“…A few works exploit heterogeneous computing resources [19], [20] such as GPUs/FPGAs to accelerate data deduplication. CROCUS [19] orchestrates CPUs and GPUs to enhance the throughput of deduplication systems. REDUP [21] provides a deduplication-aware hierarchical caching mechanism to accelerate I/O operations for machine learning frameworks.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Accelerating Content-Defined Chunking for Data Deduplication Based on Speculative Jump

Jin,

Liu,

et al. 2023

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

In data deduplication systems, chunking has a significant impact on the deduplication ratio and throughput. Existing Content-Defined Chunking (CDC) approaches exploit a sliding window to calculate rolling hashes of the input data stream byteby-byte, and then determine chunk cut-points if the rolling hash satisfies a given cut-condition. Since previous CDC approaches are extremely costly, it often significantly degrades the throughput of data deduplication systems. In this paper, we argue that calculating and checking the rolling hashes byte-by-byte is unnecessary. To reduce the CPU overhead of CDC, we propose a jump-based chunking (JC) approach. The key idea is to introduce a jump-condition, and the sliding window can jump over a specific length of the input data stream if the rolling hashes satisfy the jump-condition. Moreover, we also explore the impact of the cut-condition and the jumpcondition on the chunk size. Our theoretic studies demonstrate the effectiveness and efficiency of JC, without compromising the deduplication ratio. Experimental results show that JC improves the throughput of chunking by about 2× on average compared with the state-of-the-art CDC approaches while still guaranteeing high deduplication ratio.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Accelerating Content-Defined Chunking for Data Deduplication Based on Speculative Jump

Jin,

Liu,

et al. 2023

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

show abstract

“…In terms of data storage, it is a distributed storage ledger; in terms of protocol, it is a decentralized consensus protocol; and in terms of economy, it is an Internet of value that improves cooperation efficiency. From the perspective of blockchain technology, blockchain 1 data storage uses hash compression, asymmetric encryption, 2 and other cryptographic principles to ensure reliability and to adopt distributed data storage 3,4 through point-to-point connections. 5 The blockchain ledger is jointly maintained by all nodes and stores data, which is based on a credible consensus mechanism.…”

Section: Introductionmentioning

confidence: 99%

Second-degree branch structure blockchain expansion model

Qiang

Song

Wang

2022

International Journal of Distributed Sensor Networks

View full text Add to dashboard Cite

The blockchain runs in a complex topological network which is affected by the principle of consensus, and data storage between nodes needs to maintain global consistency in the entire network, which causes the data storage inefficient. At the same time, the information exchange between large-scale communication node groups leads to the problems of bandwidth expropriation and excessive network load. In response to these problems, this article proposes a second-degree branch structure blockchain expansion model. First, a ternary storage structure is established. Data use the way of fully integrated storage, multi-cell storage, and fully split storage, and data are classified and stored in parallel between the structures. Second, a second-degree branch chain model is constructed. The main chain forks into multiple sub-chains, and a free competition chain structure and a Z-type chain structure are defined; a two-way rotation mechanism is introduced to realize the integration and transition between chain structures. Finally, a set of malicious attacks is simulated to realize the security constraints of the blockchain, to verify the security of the second-degree branch chain model. Experiment shows that the second-degree branch structure expansion model proposed in this article has great advantages in data storage efficiency and network load.

show abstract

“…Since cluster-wide deduplication has been adopted in the general storage community [7], [8], [9], [10], [11], [12], [13], [14], it is necessary to extend its advantages to the ML and DL storage architectures without affecting performance. Deduplication improves the storage capacity by removing duplicate data across the cluster.…”

Section: Introductionmentioning

confidence: 99%

Accelerating ML/DL Applications with Hierarchical Caching on Deduplication Storage Clusters

Hamandawana

Khan

Kim

et al. 2021

IEEE Trans. Big Data

Self Cite

View full text Add to dashboard Cite

Large scale machine learning (ML) and deep learning (DL) platforms face challenges when integrated with deduplication enabled storage clusters. In the quest to achieve smart and efficient storage utilization, removal of duplicate data introduces bottlenecks, since deduplication alters the I/O transaction layout of the storage system. Therefore, it is critical to address such deduplication overhead for acceleration of ML/DL computation in deduplication storage. Existing state of the art ML/DL storage solutions such as Alluxio and AutoCache adopt non deduplication-aware caching mechanisms, which lacks the much needed performance boost when adopted in deduplication enabled ML/DL clusters. In this paper, we introduce Redup, which eliminates the performance drop caused by enabling deduplication in ML/DL storage clusters. At the core, is a Redup Caching Manager (RDCM), composed of a 2-tier deduplication layoutaware caching mechanism. The RDCM provides an abstraction of the underlying deduplication storage layout to ML/DL applications and provisions a decoupled acceleration of object reconstruction during ML/DL read operations. Our Redup evaluation shows negligible performance drop in ML/DL training performances as compared to a cluster without deduplication, whilst significantly outperforming Alluxio and AutoCache in terms of various performance metrics.

show abstract

Crocus: Enabling Computing Resource Orchestration for Inline Cluster-Wide Deduplication on Scalable Storage Systems

Cited by 10 publications

References 25 publications

Accelerating Content-Defined Chunking for Data Deduplication Based on Speculative Jump

Accelerating Content-Defined Chunking for Data Deduplication Based on Speculative Jump

Second-degree branch structure blockchain expansion model

Accelerating ML/DL Applications with Hierarchical Caching on Deduplication Storage Clusters

Contact Info

Product

Resources

About