Two High-Performance Alternatives to ZLIB Scientific-Data Compression

Almeida, Samuel Santos; Oliveira, Vítor; Pina, António Manuel Silva; Melle‐Franco, Manuel

doi:10.1007/978-3-319-09147-1_45

Cited by 6 publications

(3 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The zlib algorithm [ 30 ] is a widely used and versatile compression algorithm that enables lossless data compression. It was developed by Jean-Loup Gailly and Mark Adler and is an integral part of the widely used zlib library.…”

Section: Compression Algorithmsmentioning

confidence: 99%

Efficient Data Management in Agricultural IoT: Compression, Security, and MQTT Protocol Analysis

Has,

Kreković,

Kušek

et al. 2024

Sensors

View full text Add to dashboard Cite

The integration of Internet of Things (IoT) technology into agriculture has revolutionized farming practices by using connected devices and sensors to optimize processes and facilitate sustainable execution. Because most IoT devices have limited resources, the vital requirement to efficiently manage data traffic while ensuring data security in agricultural IoT solutions creates several challenges. Therefore, it is important to study the data amount that IoT protocols generate for resource-constrained devices, as it has a direct impact on the device performance and overall usability of the IoT solution. In this paper, we present a comprehensive study that focuses on optimizing data transmission in agricultural IoT solutions with the use of compression algorithms and secure technologies. Through experimentation and analysis, we evaluate different approaches to minimize data traffic while protecting sensitive agricultural data. Our results highlight the effectiveness of compression algorithms, especially Huffman coding, in reducing data size and optimizing resource usage. In addition, the integration of encryption techniques, such as AES, provides the security of the transmitted data without incurring significant overhead. By assessing different communication scenarios, we identify the most efficient approach, a combination of Huffman encoding and AES encryption, to strike a balance between data security and transmission efficiency.

show abstract

Section: Compression Algorithmsmentioning

confidence: 99%

Efficient Data Management in Agricultural IoT: Compression, Security, and MQTT Protocol Analysis

Has,

Kreković,

Kušek

et al. 2024

Sensors

View full text Add to dashboard Cite

show abstract

“…All data types in the NumPy array protocol are acceptable for zarray encoding. There exists a plethora of literature that details compression algorithm performance and benchmark test results that aid choosing appropriate compression schemes for a particular use case (Donoho 1993;Alted 2010;Almeida et al 2014;Wang et al 2015;Kuhn et al 2016).…”

Section: B) Zarrmentioning

confidence: 99%

Using Cloud Computing to Analyze Model Output Archived in Zarr Format

Gowan

Horel

Jacques

et al. 2022

Journal of Atmospheric and Oceanic Technology

View full text Add to dashboard Cite

Numerical weather prediction centers rely on the GRIdded Binary Second Edition (GRIB2) file format to efficiently compress and disseminate model output as two-dimensional grids. User processing time and storage requirements are high if many GRIB2 files with size O(100 MB) need to be accessed routinely. We illustrate one approach to overcome such bottlenecks by reformatting GRIB2 model output from the High-Resolution Rapid Refresh (HRRR) model of the National Centers for Environmental Prediction to a cloud-optimized storage type, Zarr. Archives of the original HRRR GRIB2 files and the resulting Zarr stores on Amazon Web Services (AWS) Simple Storage Service (S3) are available publicly through the Amazon Sustainability Data Initiative. Every hour, the HRRR model produces 18 or 48 hourly GRIB2 surface forecast files of size O(100 mb). To simplify access to the grids in the surface files, we reorganize the HRRR model output for each variable and vertical level into Zarr stores of size O(1 MB), with chunks O(10 kb) containing all forecast lead times for 150 ×150 grid point subdomains. Open-source libraries provide efficient access to the compressed Zarr stores using cloud or local computing resources. The HRRR-Zarr approach is illustrated for common applications of sensible weather parameters, including real-time alerts for high-impact situations and retrospective access to output from 100’s-1000’s of model runs. For example, time series of surface pressure forecast grids can be accessed using AWS cloud computing resources approximately 40 times faster from the HRRR-Zarr store than from the HRRR-GRIB2 archive.

show abstract

“…Thus data compression has the potential to be combined with failure detection and proactive checkpointing to further improve the system efficiency. In [32], authors conducted a thorough comparative study between popular compressors ZLIB, LZ4, and FPC. As another example, data compression was also used in reducing the MPI trace size, as shown in [33].…”

Section: Related Workmentioning

confidence: 99%

Dynamic Virtual Chunks: On Supporting Efficient Accesses to Compressed Scientific Data

Zhao

Qiao

Yin

et al. 2016

IEEE Trans. Serv. Comput.

View full text Add to dashboard Cite

Data compression could ameliorate the I/O pressure of data-intensive scientific applications. Unfortunately, the conventional wisdom of naively applying data compression to the file or block brings the dilemma between efficient random accesses and high compression ratios. File-level compression barely supports efficient random accesses to the compressed data: any retrieval request need trigger the decompression from the beginning of the compressed file. Block-level compression provides flexible random accesses to the compressed blocks, but introduces extra overhead when applying the compressor to each and every block that results in a degraded overall compression ratio. This paper extends our prior work that introduces virtual chunks offering efficient random accesses to the compressed scientific data without sacrificing the compression ratio. Virtual chunks are logical blocks pointed at by appended references without breaking the physical continuity of the file content. These references allow the decompression to start from an arbitrary position (efficient random accesses), while no per-block overhead is introduced because the file's physical entirety is retained (high compression ratio). One limitation of virtual chunk is it only supports static references. This paper presents the algorithms, analysis, and evaluations of dynamic virtual chunks to deal with the cases where the references are updated dynamically.

show abstract

Two High-Performance Alternatives to ZLIB Scientific-Data Compression

Cited by 6 publications

References 12 publications

Efficient Data Management in Agricultural IoT: Compression, Security, and MQTT Protocol Analysis

Efficient Data Management in Agricultural IoT: Compression, Security, and MQTT Protocol Analysis

Using Cloud Computing to Analyze Model Output Archived in Zarr Format

Dynamic Virtual Chunks: On Supporting Efficient Accesses to Compressed Scientific Data

Contact Info

Product

Resources

About