Polycomp: Efficient and configurable compression of astronomical timelines

Abstract

This paper describes the implementation of polycomp, a open-sourced, publicly available program for compressing one-dimensional data series in tabular format. The program is particularly suited for compressing smooth, noiseless streams of data like pointing information, as one of the algorithms it implements applies a combination of least squares polynomial fitting and discrete Chebyshev transforms that is able to achieve a compression ratio C r up to ≈ 40 in the examples discussed in this work. This performance comes at the expense of a loss of information, whose upper bound is configured by the user. I show two areas in which the usage of polycomp is interesting. In the first example, I compress the ephemeris table of an astronomical object (Ganymede), obtaining C r ≈ 20, with a compression error on the x, y, z coordinates smaller than 1 m. In the second example, I compress the publicly available timelines recorded by the Low Frequency Instrument (LFI), an array of microwave radiometers onboard the ESA Planck spacecraft. The compression reduces the needed storage from ∼ 6.5 TB to ≈ 0.75 TB (C r ≈ 9), thus making them small enough to be kept in a portable hard drive.

Enabling Near Real-Time Remote Search for Fast Transient Events with Lossy Data Compression

“…It has been used to minimise the volume of astronomical data since the 1970s, and has continued to be developed and used ever since (e.g. Labrum, McLean, & Wild 1975; White & Percival 1994; Pence et al 2000; Pence, Seaman, & White 2011; Tomasi 2016). Two main categories of compression exist: lossless and lossy compression.…”