Fast Lossless Compression of Scientific Floating-Point Data

Ratanaworabhan, Paruj; Ke, Jianhong; Burtscher, Martin

doi:10.1109/dcc.2006.35

Cited by 116 publications

(88 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…1 There were 51 snapshots, each comprising an array of 1020 × 1020 floating point numbers. These data represent a differentiated quantity-vorticity-which is obtained by taking the curl of velocity.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

An Adaptive Sub-sampling Method for In-memory Compression of Scientific Data

Unat

Hromadka

Baden

2009

2009 Data Compression Conference

View full text Add to dashboard Cite

A current challenge in scientific computing is how to curb the growth of simulation datasets without losing valuable information. While wavelet based methods are popular, they require that data be decompressed before it can analyzed, for example, when identifying time-dependent structures in turbulent flows. We present Adaptive Coarsening, an adaptive subsampling compression strategy that enables the compressed data product to be directly manipulated in memory without requiring costly decompression. We demonstrate compression factors of up to 8 in turbulent flow simulations in three dimensions. Our compression strategy produces a non-progressive multiresolution representation, subdividing the dataset into fixed sized regions and compressing each region independently.

show abstract

Section: Resultsmentioning

confidence: 99%

“…Due to the presence of noise in floating point data, lossless compression [1], [2] offers only modest improvement, and lossy compression is employed in practice. The majority of the published compression techniques involve data visualization [3], [4] and include multiresolution techniques such as wavelets [5], [6].…”

Section: Introductionmentioning

confidence: 99%

An Adaptive Sub-sampling Method for In-memory Compression of Scientific Data

Unat

Hromadka

Baden

2009

2009 Data Compression Conference

View full text Add to dashboard Cite

show abstract

“…Compression Algorithms/Techniques 7Zip [1], [19] Back-End: LZMA (Default), LZMA2, PPMd, BZip2, DEFLATE BZip2 [19] Uses Burrows-Wheeler Block Transformation (BWT) and Huffman GZip [19] Based on DEFLATE (LZ77 and Huffman) PPMd [22], [7], [20] Based on Prediction by Partial Matching (PPM), Adaptive Statistical Technique using Context Modelling and Prediction PPMVC [22], [12], [20] Based on PPM with Variable-Length Contexts ZPAQ [24], [17] An Extension to PPM, Back-End: LZ77 (Default), BWT Data-specific transforms have been developed for this purpose for different types of data. For example, to transform textual data, WRT was developed by Skibiński [21], [12], [20] to pre-process English language text based on matching words in the dictionary and replacing them with codewords.…”

Section: Compressorsmentioning

confidence: 99%

“…Dictionary-based approaches involve the substitution of words or phrases with their corresponding indices in the dictionary. The Lempel-Ziv compression schemes which include LZ77, LZ78, LZW and LZMA to name a few, are all dictionary-based approaches [4], [19], [20].…”

Section: General-purpose Data Compressorsmentioning

confidence: 99%

A Fully Reversible Data Transform Technique Enhancing Data Compression of SMILES Data

Scanlon

Ridley

2013

Availability, Reliability, and Security in Information Systems and HCI

View full text Add to dashboard Cite

Abstract. The requirement to efficiently store and process SMILES data used in Chemoinformatics creates a demand for efficient techniques to compress this data. General-purpose transforms and compressors are available to transform and compress this type of data to a certain extent, however, these techniques are not specific to SMILES data. We develop a transform specific to SMILES data that can be used alongside other general-purpose compressors as a preprocessor and post-processor to improve the compression of SMILES data. We test our transform with six other general-purpose compressors and also compare our results with another transform on our SMILES data corpus, we also compare our results with untransformed data.Keywords: SMILES, Data Transform, Data Compression. IntroductionThe Simplified Molecular Input Line Entry System (SMILES) language was developed to represent two-dimensional molecular structures in a concise and compact way allowing for storage and processing improvements. General-purpose compressors allow for further reductions in storage and processing costs [23], [8].With the continuous expansion of chemical databases [15] and the need for efficient storage and searching of molecular structure representations, such as SMILES [23], [8], storage and processing costs of these representations need to be further improved.Data can be transformed by exploiting the specific information contained in the data to its advantage. Transformed data can be used alongside general-purpose compression techniques to further improve compression results [21], [4], [20].We make the following contributions in this paper:• We present our SMILES-specific transform designed to enhance the compression of SMILES data when used with other general-purpose compressors.• We provide results from using general-purpose compression techniques on a breakdown of different SMILES transform scenarios and a combination of techniques used in our SMILES transforms.

show abstract

“…A rather different kind of method involves compressing a stream of floating point values using prediction and leading zero count compression [16]. Although this method could be adapted to our application scenario with some modifications, in this study we focus only on the quantization-based compression methods.…”

Section: Introductionmentioning

confidence: 99%

Robust Decentralized Mean Estimation with Limited Communication

Danner

Jelasity

2018

Euro-Par 2018: Parallel Processing

View full text Add to dashboard Cite

Abstract. Mean estimation, also known as average consensus, is an important computational primitive in decentralized systems. When the average of large vectors has to be computed, as in distributed data mining applications, reducing the communication cost becomes a key design goal. One way of reducing communication cost is to add dynamic stateful encoders and decoders to traditional mean estimation protocols. In this approach, each element of a vector message is encoded in a few bits (often only one bit) and decoded by the recipient node. However, due to this encoding and decoding mechanism, these protocols are much more sensitive to benign failure such as message drop and message delay. Properties such as mass conservation are harder to guarantee. Hence, known approaches are formulated under strong assumptions such as reliable communication, atomic non-overlapping transactions or even full synchrony. In this work, we propose a communication efficient algorithm that supports known codecs even if transactions overlap and the nodes are not synchronized. The algorithm is based on push-pull averaging, with novel features to support fault tolerance and compression. As an independent contribution, we also propose a novel codec, called the pivot codec. We demonstrate experimentally that our algorithm improves the performance of existing codecs and the novel pivot codec dominates the competing codecs in the scenarios we studied.

show abstract

Fast Lossless Compression of Scientific Floating-Point Data

Cited by 116 publications

References 7 publications

An Adaptive Sub-sampling Method for In-memory Compression of Scientific Data

An Adaptive Sub-sampling Method for In-memory Compression of Scientific Data

A Fully Reversible Data Transform Technique Enhancing Data Compression of SMILES Data

Robust Decentralized Mean Estimation with Limited Communication

Contact Info

Product

Resources

About