Enabling Near Real-Time Remote Search for Fast Transient Events with Lossy Data Compression

Abstract

We present a systematic evaluation of JPEG2000 (ISO/IEC 15444) as a transport data format to enable rapid remote searches for fast transient events as part of the Deeper Wider Faster program (DWF). DWF uses ∼20 telescopes from radio to gamma-rays to perform simultaneous and rapid-response follow-up searches for fast transient events on millisecond-to-hours timescales. DWF search demands have a set of constraints that is becoming common amongst large collaborations. Here, we focus on the rapid optical data component of DWF led by the Dark Energy Camera (DECam) at Cerro Tololo Inter-American Observatory (CTIO). Each DECam image has 70 total coupled-charged devices saved as a ∼1.2 gigabyte FITS file. Near real-time data processing and fast transient candidate identifications -in minutes for rapid follow-up triggers on other telescopes -requires computational power exceeding what is currently available on-site at CTIO. In this context, data files need to be transmitted rapidly to a foreign location for supercomputing post-processing, source finding, visualization and analysis. This step in the search process poses a major bottleneck, and reducing the data size helps accommodate faster data transmission. To maximise our gain in transfer time and still achieve our science goals, we opt for lossy data compression -keeping in mind that raw data is archived and can be evaluated at a later time. We evaluate how lossy JPEG2000 compression affects the process of finding transients, and find only a negligible effect for compression ratios up to ∼25:1. We also find a linear relation between compression ratio and the mean estimated data transmission speed-up factor. Adding highly customized compression and decompression steps to the science pipeline considerably reduces the transmission time -validating its introduction to the DWF science pipeline and enabling science that was otherwise too difficult with current technology.

Follow Up of GW170817 and Its Electromagnetic Counterpart by Australian-Led Observing Programmes

“…As a result, DWF is on-source before, during, and after fast transients and has been in full operation since 2016. Moreover, DWF performs real-time (seconds) supercomputing data analysis and transient identification (Andreoni et al 2017a; Vohl et al 2017; Meade et al 2017) and triggers rapid-response, conventional ToO, and long-term spectroscopy and imaging with our network of 1–10-m class telescopes. DWF operates several weeks a year and was not on sky during this GW event.…”

Mary, a Pipeline to Aid Discovery of Optical Transients

“…The DWF programme (Cooke et al, in preparation; Vohl et al 2017; Andreoni et al 2017a, 2017b) organises and conducts several components to detect and study fast transients.…”

“…In this section, we tested the timing performance of the Mary pipeline, without accounting for the data transfer time. The data compression and transfer from CTIO (in Chile) to the Swinburne supercomputer (in Australia) affects the near real-time observations in a way extensively discussed in two separate papers (Cooke et al in preparation; Vohl et al, 2017) specifically dedicated to the DWF programme. In particular, Vohl et al present the custom compression code based on JPEG2000 standard for the compression of DECam data that significantly reduces the time needed to transfer the images from CTIO to the Swinburne supercomputer in Australia.…”

“…The DWF programme (Cooke et al, in preparation;Vohl et al 2017;Andreoni et al 2017aAndreoni et al , 2017b organises and conducts several components to detect and study fast transients . The first component coordinates simultaneous observations of the same targeted field, among a list of target fields, with multiple facilities operating at different wavelengths.…”

Use cases of lossy compression for floating-point data in scientific data sets

“…Reducing data stream intensity is a typical need of large-scale instruments such as accelerators and telescopes, and several lossy compression techniques have been proposed or tested specifically for these instruments (Nicolaucig et al, 2003; Offringa, 2016; Patauner, 2011; Peters and Kitaeff, 2014; Röhrich and Vestø, 2006; Vohl et al, 2015, 2017). In this section, we focus on light source facilities such as the Advanced Photon Source (APS) (Fornek, 2017) (a synchrotron-radiation light source research facility at Argonne) and the Linear Coherent Light Source (LCLS) (Marcus et al, 2015) (a free-electron laser facility located at SLAC).…”