Alexandria: A Proof-of-Concept Implementation and Evaluation of Generalised Data Deduplication

Abstract

The amount of data generated worldwide is expected to grow from 33 to 175 ZB by 2025 [1] in part driven by the growth of Internet of Things (IoT) and cyber-physical systems (CPS). To cope with this enormous amount of data, new edge (and cloud) storage techniques must be developed. Generalised Data Deduplication (GDD) is a new paradigm for reducing the cost of storage by systematically identifying near identical data chunks, storing their common component once, and a compact representation of the deviation to the original chunk for each chunk. This paper presents a system architecture for GDD and a proof-of-concept implementation. We evaluated the compression gain of Generalised Data Deduplication using three data sets of varying size and content and compared to the performance of the EXT4 and ZFS file systems, where the latter employs classic deduplication. We show that Generalised Data Deduplication provide up to 16.75% compression gain compared to both EXT4 and ZFS with data sets with less than 5 GB of data.

A Priority-Based Multicast Flow Scheduling Method for a Collaborative Edge Storage Datacenter Network

“…As we can see in Fig. 2, the path from ESDC1 to ESDC2 may be (1, 4, 2), (1,5,2), (1,4,6,2), etc. We call these paths sub-path set1.…”

“…With the rapid development of the Internet of Everything (IoE), the growing data storage requirement is posing a complex technical challenge for the IT community. According to the white book Data-Age-2025 [1] from the International Data Corporation (IDC), the global data scale will grow from 33 zettabytes in 2018 to 175 by 2025. At the same time, the emergence of many new applications such as unmanned driving and smart cities will lead to a greater demand for reducing data transmission delay [2].…”

What If Keys Are Leaked? towards Practical and Secure Re-Encryption in Deduplication-Based Cloud Storage

“…Yao et al [60] proposed a Hierarchical Privilege-Based Predicate Encryption (HPBPE) scheme based on the hierarchical encryption, which introduces a deduplication provider to check duplication based on user dynamic privileges of users without revealing any information about privilege to the cloud server. To further enhance the effectiveness of deduplication, some generalized deduplication schemes [61][62][63] are proposed by detecting and removing "similar" data rater than "identical" data using transformation functions, e.g., the Hamming code and the Revealing Encryption (RE), in traditional deduplication schemes.…”

ZipLine

“…Programmable network switches have sparked a lot of interest in academia and industry. In particular, the P4 language allows implementing novel tools, e.g., real-time network telemetry and analytics [22,33], and accelerating existing applications, e.g., stream processing [17]. One of the reasons for P4's current momentum is that it allows easy verification of in-network programs [20,30].…”

“…Although originally considered for large scale data storage [21,32], GD has been adapted to multi-source data compression protocols [15] and file compression for time-series data [35,37]. This has resulted in lightweight, online compression mechanisms suitable to the Internet of Things (IoT) an file compressors with excellent random access properties.…”