Dynamic Virtual Chunks: On Supporting Efficient Accesses to Compressed Scientific Data

Abstract

Data compression could ameliorate the I/O pressure of data-intensive scientific applications. Unfortunately, the conventional wisdom of naively applying data compression to the file or block brings the dilemma between efficient random accesses and high compression ratios. File-level compression barely supports efficient random accesses to the compressed data: any retrieval request need trigger the decompression from the beginning of the compressed file. Block-level compression provides flexible random accesses to the compressed blocks, but introduces extra overhead when applying the compressor to each and every block that results in a degraded overall compression ratio. This paper extends our prior work that introduces virtual chunks offering efficient random accesses to the compressed scientific data without sacrificing the compression ratio. Virtual chunks are logical blocks pointed at by appended references without breaking the physical continuity of the file content. These references allow the decompression to start from an arbitrary position (efficient random accesses), while no per-block overhead is introduced because the file's physical entirety is retained (high compression ratio). One limitation of virtual chunk is it only supports static references. This paper presents the algorithms, analysis, and evaluations of dynamic virtual chunks to deal with the cases where the references are updated dynamically.

Hesitant Fuzzy Entropy-Based Opportunistic Clustering and Data Fusion Algorithm for Heterogeneous Wireless Sensor Networks

“…Possible future work could include HFE-based ring routing in WSNs, HFE-based heterogeneous clustering supporting sink mobility, HFE based on Cognitive Radio Sensor Networks (CRSN) etc. In addition to this, the proposed scheme can be applied to ring routing [49], big data systems [50], data compression [51] and blockchain technology [52] in terms of reliable data fusion.…”

A New Data Fusion Algorithm for Wireless Sensor Networks Inspired by Hesitant Fuzzy Entropy

“…We have shown that the proposed data fusion algorithm could improve the reliability of data compared to the state-of-the-art. In the future, we plan to integrate the proposed data fusion algorithm into other system research areas, such as big data systems [ 28 ], key-value stores [ 29 ], data compression [ 30 ], and blockchains [ 31 ].…”