Data compression in the petascale astronomy era: A GERLUMPH case study

Vohl, Dany; Fluke, C. J.; Vernardos, G.

doi:10.1016/j.ascom.2015.05.003

Cited by 12 publications

(14 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…where size o is size of the original file and size c the size of the compressed file. Recent investigations of lossy JPEG2000 compression for astronomical images (Peters & Kitaeff, 2014;Kitaeff et al, 2015;Vohl et al, 2015) show that it can lend high factors of compression while preserving scientifically important information in the data. For example, Peters & Kitaeff (2014) compressed synthetic radio astronomy data at several levels of compression, and evaluated how the loss affects the process of source finding.…”

Section: Jpeg2000 and Lossy Data Compressionmentioning

confidence: 99%

“…In the time-critical scenario of DWF, a gain in transmission time offered by data compression is only interesting if compression and decompression can be achieved quickly. To this end, Vohl et al (2015) demonstrated that KERLUMPH 5 -a multi-threaded implementation of the JPEG2000 standard -can compress and decompress large files quickly.…”

Section: Software Design Rationalementioning

confidence: 99%

See 1 more Smart Citation

Enabling Near Real-Time Remote Search for Fast Transient Events with Lossy Data Compression

Vohl

Andreoni

Cooke

et al. 2017

Publ. Astron. Soc. Aust.

View full text Add to dashboard Cite

We present a systematic evaluation of JPEG2000 (ISO/IEC 15444) as a transport data format to enable rapid remote searches for fast transient events as part of the Deeper Wider Faster program (DWF). DWF uses ∼20 telescopes from radio to gamma-rays to perform simultaneous and rapid-response follow-up searches for fast transient events on millisecond-to-hours timescales. DWF search demands have a set of constraints that is becoming common amongst large collaborations. Here, we focus on the rapid optical data component of DWF led by the Dark Energy Camera (DECam) at Cerro Tololo Inter-American Observatory (CTIO). Each DECam image has 70 total coupled-charged devices saved as a ∼1.2 gigabyte FITS file. Near real-time data processing and fast transient candidate identifications -in minutes for rapid follow-up triggers on other telescopes -requires computational power exceeding what is currently available on-site at CTIO. In this context, data files need to be transmitted rapidly to a foreign location for supercomputing post-processing, source finding, visualization and analysis. This step in the search process poses a major bottleneck, and reducing the data size helps accommodate faster data transmission. To maximise our gain in transfer time and still achieve our science goals, we opt for lossy data compression -keeping in mind that raw data is archived and can be evaluated at a later time. We evaluate how lossy JPEG2000 compression affects the process of finding transients, and find only a negligible effect for compression ratios up to ∼25:1. We also find a linear relation between compression ratio and the mean estimated data transmission speed-up factor. Adding highly customized compression and decompression steps to the science pipeline considerably reduces the transmission time -validating its introduction to the DWF science pipeline and enabling science that was otherwise too difficult with current technology.

show abstract

Section: Jpeg2000 and Lossy Data Compressionmentioning

confidence: 99%

Section: Software Design Rationalementioning

confidence: 99%

Enabling Near Real-Time Remote Search for Fast Transient Events with Lossy Data Compression

Vohl

Andreoni

Cooke

et al. 2017

Publ. Astron. Soc. Aust.

View full text Add to dashboard Cite

show abstract

“…If the development of a theoretical model for the compression is too complex, the most common approach is to estimate the error induced on the results of the data analysis when compressed data are used instead of the original uncompressed ones. For a few examples of the latter approach, see e.g., Vohl et al (2015); Löptien et al (2016).…”

Section: Inputmentioning

confidence: 99%

Polycomp: Efficient and configurable compression of astronomical timelines

Tomasi

2016

Astronomy and Computing

View full text Add to dashboard Cite

This paper describes the implementation of polycomp, a open-sourced, publicly available program for compressing one-dimensional data series in tabular format. The program is particularly suited for compressing smooth, noiseless streams of data like pointing information, as one of the algorithms it implements applies a combination of least squares polynomial fitting and discrete Chebyshev transforms that is able to achieve a compression ratio C r up to ≈ 40 in the examples discussed in this work. This performance comes at the expense of a loss of information, whose upper bound is configured by the user. I show two areas in which the usage of polycomp is interesting. In the first example, I compress the ephemeris table of an astronomical object (Ganymede), obtaining C r ≈ 20, with a compression error on the x, y, z coordinates smaller than 1 m. In the second example, I compress the publicly available timelines recorded by the Low Frequency Instrument (LFI), an array of microwave radiometers onboard the ESA Planck spacecraft. The compression reduces the needed storage from ∼ 6.5 TB to ≈ 0.75 TB (C r ≈ 9), thus making them small enough to be kept in a portable hard drive.

show abstract

“…1 . The relation between 2 1 and̃2 1 is defined by relation (2). Orthogonality in defined vector space could be written as condition…”

Section: Karhunen-loève Transformmentioning

confidence: 99%

“…This data is necessary for archiving and distribution among servers and users. Therefore, searching for suitable compression standard is still rather important task [2,3]. There are many compression standards matched to the human visual system (HVS) for multimedia purposes [4].…”

Section: Introductionmentioning

confidence: 99%

The Impact of KLT Coder on the Image Distortion in Astronomy

Páta

2015

Advances in Astronomy

View full text Add to dashboard Cite

Presented paper is devoted to the application of Karhunen-Loève transform (KLT) for compression and to study of KLT impact on the image distortion in astronomy. This transform is an optimal fit for images with Gaussian probability density function in order to minimize the root mean square error (RMSE). The main part of the encoder is proposed in relation to statistical image properties. Selected astronomical image processing algorithms are used for the encoder testing. The astrometry and point spread function distortion are selected as the most important criteria. The results are compared with JPEG2000 standard. The KLT encoder provides better results from the RMSE point of view. These results are promising and show the novel approach to the design of lossy image compression algorithms and also suitability for algorithms of image data structuring for retrieving, transfer, and distribution.

show abstract

Data compression in the petascale astronomy era: A GERLUMPH case study

Cited by 12 publications

References 43 publications

Enabling Near Real-Time Remote Search for Fast Transient Events with Lossy Data Compression

Enabling Near Real-Time Remote Search for Fast Transient Events with Lossy Data Compression

Polycomp: Efficient and configurable compression of astronomical timelines

The Impact of KLT Coder on the Image Distortion in Astronomy

Contact Info

Product

Resources

About