Characterizing the efficiency of data deduplication for big data storage management

Zhou, Ruijin; Liu, Ming; Li, Tao

doi:10.1109/iiswc.2013.6704674

Cited by 37 publications

(16 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On ZFS file system [42] with inline deduplication, the average CPU usage could boost from 20% to 60% due to heavy computation and indexing. The inline deduplication causes the storage system to consume 10% more power on average [44].…”

Section: Backgroundsmentioning

confidence: 99%

Hope

Liu

et al. 2016

Proceedings of the 2016 International Conference on Supercomputing

Self Cite

View full text Add to dashboard Cite

The functional scope of today's software-defined data centers (SDDC) has expanded to such an extent that servers face a growing amount of critical background operational tasks like load monitoring, logging, migration, and duplication, etc. These ancillary operations, which we refer to as management operations, often nibble the stringent data center power envelope and exert a tremendous amount of pressure on front-end user tasks. However, existing power capping, peak shaving, and time shifting mechanisms mainly focus on managing data center power demand at the "macro level" -they do not distinguish ancillary background services from user tasks, and therefore often incur significant performance degradation and energy overhead.In this study we explore "micro-level" power management in SDDC: tuning a specific set of critical loads for the sake of overall system efficiency and performance. Specifically, we look at management operations that can often lead to resource contention and energy overhead in an IaaS SDDC. We assess the feasibility of this new power management paradigm by characterizing the resource and power impact of various management operations. We propose HOPE, a new system optimization framework for eliminating the potential efficiency bottleneck caused by the management operations in the SDDC. HOPE is implemented on a customized OpenStack cloud environment with heavily instrumented power infrastructure. We thoroughly validate HOPE models and optimization efficacy under various user workload scenarios. Our deployment experiences show that the proposed technique allows SDDC to reduce energy consumption by 19%, reduce management operation execution time by 25.4%, and in the meantime improve workload performance by 30%.

show abstract

Section: Backgroundsmentioning

confidence: 99%

Hope

Liu

et al. 2016

Proceedings of the 2016 International Conference on Supercomputing

Self Cite

View full text Add to dashboard Cite

show abstract

“…Three main reasons for data redundancy are: (1) addition of nodes, (2) expansion of datasets, and (3) data replication. The addition of a single virtual machine (VM) brings around 97% more redundancy, and the growth in large datasets comes with 47% redundant data points [13]. In addition, the storage mechanism for maximum data availability (also called data replication) brings 100% redundancy at the cluster level.…”

Section: Data Deduplication (Redundancy Elimination)mentioning

confidence: 99%

“…In contrast, global deduplication can achieve maximum redundancy but compromises on the hashing overheads. In addition, finegrained deduplication is not suitable for big datasets especially in streaming data environments [13].…”

Section: Open Research Issuesmentioning

confidence: 99%

“…The I/O latency and extra computational overhead of cluster-level data deduplication are among the key challenges. The authors in [13] characterized the deduplication schemes in terms of energy impact and performance overhead. The authors outlined three sources of redundancy in cluster environment including: (1) the deployment of additional nodes in the cluster, (2) the expansion of big datasets, and (3) the usage of replication mechanisms.…”

Section: Open Research Issuesmentioning

confidence: 99%

“…These techniques usually work at post-data collection phases. Similarly, cluster deduplication and redundancy elimination algorithms that remove duplicated data for efficient data processing and useful knowledge discovery are primarily post-data collection methods [12][13][14][15]. Recently, the network theory concepts have also been employed for big data reduction [16][17][18].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Big Data Reduction Methods: A Survey

et al. 2016

View full text Add to dashboard Cite

Research on big data analytics is entering in the new phase called fast data where multiple gigabytes of data arrive in the big data systems every second. Modern big data systems collect inherently complex data streams due to the volume, velocity, value, variety, variability, and veracity in the acquired data and consequently give rise to the 6Vs of big data. The reduced and relevant data streams are perceived to be more useful than collecting raw, redundant, inconsistent, and noisy data. Another perspective for big data reduction is that the million variables big datasets cause the curse of dimensionality which requires unbounded computational resources to uncover actionable knowledge patterns. This article presents a review of methods that are used for big data reduction. It also presents a detailed taxonomic discussion of big data reduction methods including the network theory, big data compression, dimension reduction, redundancy elimination, data mining, and machine learning methods. In addition, the open research issues pertinent to the big data reduction are also highlighted.

show abstract

Convergent encryption enabled secure data deduplication algorithm for cloud environment

Ahmad,

Arif,

Ahmad

et al. 2024

Concurrency and Computation

View full text Add to dashboard Cite

SummaryThe exponential growth of data poses a critical challenge for cloud storage systems. Redundant data consumes valuable storage space and increases infrastructure costs. Data deduplication, a technique for eliminating duplicate data copies, offers a promising solution. However, existing deduplication techniques often compromise data security, especially when dealing with encrypted data. This paper proposes a novel approach that merges convergent encryption (CE) with data deduplication. CE leverages user data itself to generate unique encryption keys, enabling secure deduplication on encrypted data. We analyze existing literature on secure data deduplication and categorize various techniques using UML activity diagrams. We then present our proposed CE‐based deduplication system, outlining its functionalities through UML diagrams. This research contributes to the field of secure data storage by proposing a novel and secure deduplication approach. By demonstrating its efficiency and security benefits, this work paves the way for more efficient and secure cloud storage solutions. Finally, we demonstrate the system's effectiveness through a comparative analysis, highlighting its potential to significantly improve storage efficiency while maintaining data security.

show abstract

Characterizing the efficiency of data deduplication for big data storage management

Cited by 37 publications

References 6 publications

Hope

Hope

Big Data Reduction Methods: A Survey

Convergent encryption enabled secure data deduplication algorithm for cloud environment

Contact Info

Product

Resources

About