A compression scheme for radio data in high performance computing

Masui, Kiyoshi; Amiri, M.; Connor, Liam; Deng, Meiling; Fandino, Mateus; Höfer, Carolin; Halpern, M.; Hanna, D.; Hincks, Adam D.; Hinshaw, G.; Parra, John de la; Newburgh, Laura; Shaw, J. Richard; Vanderlinde, K.

doi:10.1016/j.ascom.2015.07.002

Cited by 45 publications

(47 citation statements)

References 23 publications

(26 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, most results will be presented in as general form as possible to facilitate extensions. The covariance of the visibilities is (Kulkarni 1989;Masui et al 2015):…”

Section: Antennas and Visibilitiesmentioning

confidence: 99%

Algorithms for FFT Beamforming Radio Interferometers

Masui

Shaw

et al. 2019

ApJ

Self Cite

View full text Add to dashboard Cite

Radio interferometers consisting of identical antennas arranged on a regular lattice permit fast Fourier transform beamforming, which reduces the correlation cost from O(n 2 ) in the number of antennas to O(n log n). We develop a formalism for describing this process and apply this formalism to derive a number of algorithms with a range of observational applications. These include algorithms for forming arbitrarily pointed tied-array beams from the regularly spaced Fourier-transform formed beams, sculpting the beams to suppress sidelobes while only losing percentlevel sensitivity, and optimally estimating the position of a detected source from its observed brightness in the set of beams. We also discuss the effect that correlations in the visibility-space noise, due to cross-talk and sky contributions, have on the optimality of Fourier transform beamforming, showing that it does not strictly preserve the sky information of the n 2 correlation, even for an idealized array. Our results have applications to a number of upcoming interferometers, in particular the Canadian Hydrogen Intensity Mapping Experiment-Fast Radio Burst (CHIME/FRB) project.

show abstract

“…However, most results will be presented in as general form as possible to facilitate extensions. The covariance of the visibilities is (Kulkarni 1989;Masui et al 2015):…”

Section: Antennas and Visibilitiesmentioning

confidence: 99%

Algorithms for FFT Beamforming Radio Interferometers

Masui

Shaw

et al. 2019

ApJ

Self Cite

View full text Add to dashboard Cite

show abstract

“…The public registry for HDF5 filters 1 currently lists 21 data transformations, most of them compression-related. Each HDF5 file is evaluated with and without the shuffle filter, zlib/gzip, lzf, MAFISC [22] with LZMA, szip [20], Bitshuffle [30] with LZ4, zstd, and the full Blosc [1] compression suite, again all with various parameter values.…”

Section: Data Representationmentioning

confidence: 99%

Waveform Signal Entropy and Compression Study of Whole-Building Energy Datasets

Kriechbaumer

Jorde

Jacobsen

2019

Proceedings of the Tenth ACM International Conference on Future Energy Systems

View full text Add to dashboard Cite

Electrical energy consumption has been an ongoing research area since the coming of smart homes and Internet of Things devices. Consumption characteristics and usages profiles are directly influenced by building occupants and their interaction with electrical appliances. Extracted information from these data can be used to conserve energy and increase user comfort levels. Data analysis together with machine learning models can be utilized to extract valuable information for the benefit of occupants themselves, power plants, and grid operators. Public energy datasets provide a scientific foundation to develop and benchmark these algorithms and techniques. With datasets exceeding tens of terabytes, we present a novel study of five whole-building energy datasets with high sampling rates, their signal entropy, and how a well-calibrated measurement can have a significant effect on the overall storage requirements. We show that some datasets do not fully utilize the available measurement precision, therefore leaving potential accuracy and space savings untapped. We benchmark a comprehensive list of 365 file formats, transparent data transformations, and lossless compression algorithms. The primary goal is to reduce the overall dataset size while maintaining an easy-to-use file format and access API. We show that with careful selection of file format and encoding scheme, we can reduce the size of some datasets by up to 73%.

show abstract

“…The algorithm will also have to look for correlations in the time direction, while time is generally the slowest changing dimension. It has been shown to provide good results thought (Masui et al 2015), and a combination of these methods might provide a generic algorithm that performs near-optimal in noise-dominated as well as signal-dominated situations.…”

Section: Possible Improvements and Future Workmentioning

confidence: 99%

“…-AIPS can write visibilities as 16-bit values with uniform quantization; -The Flexible Image Transport System (FITS) file format (Wells et al 1981) allows Rice, GZIP, HCompress, and PLIO compression 2 . -For compression of integer data recorded with the CHIME telescope, Masui et al (2015) showed that those data can be compressed to 28% of its original size using linear quantization and applying the BITSHUFFLE technique before LZ77 encoding (Ziv & Lempel 1977).…”

Section: Introductionmentioning

confidence: 99%

Compression of interferometric radio-astronomical data

Offringa

2016

A&A

View full text Add to dashboard Cite

Context. The volume of radio-astronomical data is a considerable burden in the processing and storing of radio observations that have high time and frequency resolutions and large bandwidths. For future telescopes such as the Square Kilometre Array (SKA), the data volume will be even larger. Aims. Lossy compression of interferometric radio-astronomical data is considered to reduce the volume of visibility data and to speed up processing. Methods. A new compression technique named "Dysco" is introduced that consists of two steps: a normalization step, in which grouped visibilities are normalized to have a similar distribution; and a quantization and encoding step, which rounds values to a given quantization scheme using a dithering scheme. Several non-linear quantization schemes are tested and combined with different methods for normalizing the data. Four data sets with observations from the LOFAR and MWA telescopes are processed with different processing strategies and different combinations of normalization and quantization. The effects of compression are measured in image plane. Results. The noise added by the lossy compression technique acts similarly to normal system noise. The accuracy of Dysco is depending on the signal-to-noise ratio (S/N) of the data: noisy data can be compressed with a smaller loss of image quality. Data with typical correlator time and frequency resolutions can be compressed by a factor of 6.4 for LOFAR and 5.3 for MWA observations with less than 1% added system noise. An implementation of the compression technique is released that provides a Casacore storage manager and allows transparent encoding and decoding. Encoding and decoding is faster than the read/write speed of typical disks. Conclusions. The technique can be used for LOFAR and MWA to reduce the archival space requirements for storing observed data. Data from SKA-low will likely be compressible by the same amount as LOFAR. The same technique can be used to compress data from other telescopes, but a different bit-rate might be required.

show abstract

A compression scheme for radio data in high performance computing

Cited by 45 publications

References 23 publications

Algorithms for FFT Beamforming Radio Interferometers

Algorithms for FFT Beamforming Radio Interferometers

Waveform Signal Entropy and Compression Study of Whole-Building Energy Datasets

Compression of interferometric radio-astronomical data

Contact Info

Product

Resources

About