Calibration of LOFAR data on the cloud

Abstract

New scientific instruments are starting to generate an unprecedented amount of data. The Low Frequency Array (LOFAR), one of the Square Kilometre Array (SKA) pathfinders, is already producing data on a petabyte scale. The calibration of these data presents a huge challenge for final users: a) extensive storage and computing resources are required; b) the installation and maintenance of the software required for the processing is not trivial; and c) the requirements of calibration pipelines, which are experimental and under development, are quickly evolving. After encountering some limitations in classical infrastructures like dedicated clusters, we investigated the viability of cloud infrastructures as a solution.We found that the installation and operation of LOFAR data calibration pipelines is not only possible, but can also be efficient in cloud infrastructures. The main advantages were: (1) the ease of software installation and maintenance, and the availability of standard APIs and tools, widely used in the industry; this reduces the requirement for significant manual intervention, which can have a highly negative impact in some infrastructures; (2) the flexibility to adapt the infrastructure to the needs of the problem, especially as those demands change over time; (3) the on-demand consumption of (shared) resources. We found that a critical factor (also in other infrastructures) is the availability of scratch storage areas of an appropriate size. We found no significant impediments associated with the speed of data transfer, the use of virtualization, the use of external block storage, or the memory available (provided a minimum threshold is reached).Finally, we considered the cost-effectiveness of a commercial cloud like Amazon Web Services. While a cloud solution is more expensive than the operation of a large, fully-utilised cluster completely dedicated to LOFAR data reduction, we found that its costs are competitive if the number of datasets to be analysed is not high, or if the costs of maintaining a system capable of calibrating LOFAR data become high. Coupled with the advantages discussed above, this suggests that a cloud infrastructure may be favourable for many users.

Evaluating virtual hosted desktops for graphics-intensive astronomy

“…Cost-benefit analyses have been conducted in relation to the use of cloud with the SKA pathfinders, such as LOFAR 8 and CHILES 9 . Sabater et al (2017) used cloud infrastructure to run the LOFAR calibration pipeline, finding the flexibility and ad hoc availability of the cloud provided a better option than traditional on-premise HPC services. Dodson et al (2016) conducted direct comparisons of the CHILES imaging pipeline using a local cluster, a National Peak cluster (Magnus at the Pawsey Supercomputing Centre 10 , Western Australia), and cloud infrastructure from AWS.…”

Accelerating Faceting Wide-Field Imaging Algorithm with FPGA for SKA Radio Telescope as a Vast Sensor Array

“…Cloud computing is a platform with excellent computing services, including the ability to scale elastically, and it can be used to process the huge amount of data in the SKA project [21]. Traditional cloud computing service providers build large data-centers with a huge number of interdependent commodity computers with CPU as the key computing unit to handle the ever-growing challenge on performance.…”

Baseline-dependent sampling and windowing for radio interferometry: data compression, field-of-interest shaping, and outer field suppression

“…Its uv-data (visibilities), assuming 24 core stations (excluding the remote and international stations) using 244 sub-bands with 64 channels per sub-band, 4 hours observation time with a 1 s temporal resolution is predicted to be ∼8376 GB using the dual high band antenna (see LOFAR calculator 1 ). However, observations with all the LOFAR national and international stations are capable of producing data volumes of the order of petabytes (Sabater et al 2017). Survey capabilities with the future SKA (unprecedented sensitivity, resolution and bandwidth) are expected to generate data by many orders of magnitude higher than any existing radio interferometer.…”