A Comprehensive Study of the Past, Present, and Future of Data Deduplication

Xia, Wen; Jiang, Hong; Feng, Dan; Douglis, Fred; Shilane, Philip; Hua, Yu; Fu, Min; Zhang, Yucheng; Zhou, Yukun

doi:10.1109/jproc.2016.2571298

Cited by 235 publications

(85 citation statements)

References 118 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Common approaches to reduce the data footprint are data compression (see, e.g., Iverson et al 2012;Storer 1988;Xia et al 2016b), sampling (see, e.g., Herrmann 2010Lohr 2009;Mendlik and Gobiet 2016) as well as statistical methods such as cluster analysis and principal component analysis (see, e.g., Rogerson 2015).…”

Section: Related Workmentioning

confidence: 99%

GPU-Accelerated Rendering Methods to Visually Analyze Large-Scale Disaster Simulation Data

et al. 2017

View full text Add to dashboard Cite

Emerging methodologies for natural hazard risk assessments involve the execution of a multitude of different interacting simulation models that produce vast amounts of spatio-temporal datasets. This data pool is further enlarged when such simulation results are post-processed using GIS operations, for example to derive information for decisionmaking. The novel approach presented in this paper makes use of the GPU-accelerated rendering pipeline to perform such operations on-the-fly without storing any results on secondary memory and thus saving large amounts of storage space. Particularly, algorithms for three frequently used geospatial analysis methods are provided, namely for the computation of difference maps using map algebra and overlay operations, distance maps and buffers as examples for proximity analyses as well as kernel density estimation and inverse distance weighting as examples for statistical surfaces. In addition, a visualization tool is presented that integrates these methods using a node-based data flow architecture. The application of this visualization tool to the results of a real-world risk assessment methodology used in civil engineering shows that the memory footprint of post-processing datasets can be reduced at the order of terabytes. Although the technique has several limitations, most notably the reduced interoperability with conventional analysis tools, it can be beneficial for other use cases. When integrated into desktop GIS applications, for example, it can be used to quickly generate a preview of the results of complex analysis chains or it can reduce the amount of data to be transferred to web or mobile GIS applications.

show abstract

Section: Related Workmentioning

confidence: 99%

GPU-Accelerated Rendering Methods to Visually Analyze Large-Scale Disaster Simulation Data

et al. 2017

View full text Add to dashboard Cite

show abstract

“…Optimizing deduplication: Existing deduplication studies (see a complete survey [64] on deduplication) exploit workload characteristics (e.g., chunk locality [38,45,65,67] and file similarity [14,65]) to mitigate indexing overhead. For example, DDFS [67] prefetches the fingerprints of nearby chunks that are likely to be accessed together.…”

Section: Related Workmentioning

confidence: 99%

Information Leakage in Encrypted Deduplication via Frequency Analysis

Qin

Lee

et al. 2017

2017 47th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN)

View full text Add to dashboard Cite

Encrypted deduplication combines encryption and deduplication to simultaneously achieve both data security and storage efficiency. State-of-the-art encrypted deduplication systems mainly build on deterministic encryption to preserve deduplication effectiveness. However, such deterministic encryption reveals the underlying frequency distribution of the original plaintext chunks. This allows an adversary to launch frequency analysis against the ciphertext chunks and infer the content of the original plaintext chunks. In this paper, we study how frequency analysis affects information leakage in encrypted deduplication, from both attack and defense perspectives. Specifically, we target backup workloads, and propose a new inference attack that exploits chunk locality to increase the coverage of inferred chunks. We further combine the new inference attack with the knowledge of chunk sizes and show its attack effectiveness against variable-size chunks. We conduct trace-driven evaluation on both real-world and synthetic datasets and show that our proposed attacks infer a significant fraction of plaintext chunks under backup workloads. To defend against frequency analysis, we present two defense approaches, namely MinHash encryption and scrambling. Our trace-driven evaluation shows that our combined MinHash encryption and scrambling scheme effectively mitigates the severity of the inference attacks, while maintaining high storage efficiency and incurring limited metadata access overhead.

show abstract

“…Next challenge worth mentioning here is about choosing appropriate type of deduplication process. Empirical studies have shown that source deduplication solution works well, when servers have adequate resources allocated for deduplication process and key management [28] mechanism. But, normally host machines rarely can allocate large amount of resources to deduplication process.…”

Section: Challenges Foundmentioning

confidence: 99%

Data Deduplication Methods: A Review

Kaur¹,

Devgan²

2017

IJITCS

View full text Add to dashboard Cite

Abstract-The cloud storage services are used to store intermediate and persistent data generated from various resources including servers and IoT based networks. The outcome of such developments is that the data gets duplicated and gets replicated rapidly especially when large number of cloud users are working in a collaborative environment to solve large scale problems in geo-distributed networks. The data gets prone to breach of privacy and high incidence of duplication. When the dynamics of cloud services change over period of time, the ownership and proof of identity operations also need to change and work dynamically for high degree of security. In this work we will study the following concepts, methods and the schemes that can make the cloud services secure and reduce the incidence of data duplication. With the help of cryptography mathematics and to increase potential storage capacity. The proposed scheme works for deduplication of data with arithmetic key validity operations that reduce the overhead and increase the complexity of the keys so that it is hard to break the keys.

show abstract

A Comprehensive Study of the Past, Present, and Future of Data Deduplication

Cited by 235 publications

References 118 publications

GPU-Accelerated Rendering Methods to Visually Analyze Large-Scale Disaster Simulation Data

GPU-Accelerated Rendering Methods to Visually Analyze Large-Scale Disaster Simulation Data

Information Leakage in Encrypted Deduplication via Frequency Analysis

Data Deduplication Methods: A Review

Contact Info

Product

Resources

About