An Automated Scalable Framework for Distributing Radio Astronomy Processing Across Clusters and Clouds

Astronomy and Computing

Shimwell

Plaat³

et al. 2019

LOFAR is a leading aperture synthesis telescope operated in the Netherlands with stations across Europe. The LOFAR Two-meter Sky Survey (LoTSS) will produce more than 3000 14 TB data sets, mapping the entire northern sky at low frequencies. The data produced by this survey is important for understanding the formation and evolution of galaxies, supermassive black holes and other astronomical phenomena. All of the LoTSS data needs to be processed by the LOFAR Direction Independent (DI) pipeline, prefactor. Understanding the performance of this pipeline is important when trying to optimize the throughput for large projects, such as LoTSS and other deep surveys. Making a model of its completion time will enable us to predict the time taken to process large data sets, optimize our parameter choices, help schedule other LOFAR processing services, and predict processing time for future large radio telescopes. We tested the prefactor pipeline by scaling several parameters, notably number of CPUs, data size and size of calibration sky model. We present these results as a comprehensive model which will be used to predict processing time for a wide range of processing parameters. We also discover that smaller calibration models lead to significantly faster calibration times, while the calibration results do not significantly degrade in quality. Finally, we validate the model and compare predictions with production runs from the past six months, quantifying the performance penalties incurred by processing on a shared cluster. We conclude by noting the utility of the results and model for the LoTSS Survey, LOFAR as a whole and for other telescopes.

Section: Related Workmentioning

confidence: 99%

Section: Processing Setupmentioning

confidence: 99%

See 1 more Smart Citation

Scalability model for the LOFAR direction independent pipeline

Astronomy and Computing

Shimwell

Plaat³

et al. 2019

“…1. Design of the AGLOW software, incorporating Airflow, the GRID LRT package [7] and custom operators designed to integrate LOFAR software, Grid middleware and dCache storage. GRID LRT is a software package developed to parallelize single LOFAR jobs at SURFsara.…”

Section: B Aglow: Implementationmentioning

confidence: 99%

“…Additionally, many software teams in biotech develop their own in-house workflow management software [18]. * http://wlcg.web.cern.ch/ † https://toil.readthedocs.io Currently, we can parallelize a single processing step of the pipeline using the Grid LOFAR Tools (GRID LRT ‡ ) [7]. The LOFAR Surveys science cases incorporate multiple steps with inter-linked dependencies.…”

Section: Introductionmentioning

confidence: 99%

Fast and Reproducible LOFAR Workflows with AGLOW

2018 IEEE 14th International Conference on E-Science (E-Science)

Oonk

Shimwell

et al. 2018

The LOFAR radio telescope creates Petabytes of data per year. This data is important for many scientific projects. The data needs to be efficiently processed within the timespan of these projects in order to maximize the scientific impact. We present a workflow orchestration system that integrates LOFAR processing with a distributed computing platform. The system is named Automated Grid-enabled LOFAR Workflows (AGLOW). AGLOW makes it fast and easy to develop, test and deploy complex LOFAR workflows, and to accelerate them on a distributed cluster architecture. AGLOW provides a significant reduction in time for setting up complex workflows: typically, from months to days. We lay out two case studies that process the data from the LOFAR Surveys Key Science Project. We have implemented these into the AGLOW environment. We also describe the capabilities of AGLOW, paving the way for use by other LOFAR science cases. In the future, AGLOW will automatically produce multiple science products from a single dataset, serving several of the LOFAR Key Science Projects.

Pipeline Collector: Gathering performance data for distributed astronomical pipelines

Astronomy and Computing

Plaat²,

Oonk

et al. 2018

Self Cite

Modern astronomical data processing requires complex software pipelines to process ever growing datasets. For radio astronomy, these pipelines have become so large that they need to be distributed across a computational cluster. This makes it difficult to monitor the performance of each pipeline step. To gain insight into the performance of each step, a performance monitoring utility needs to be integrated with the pipeline execution. In this work we have developed such a utility and integrated it with the calibration pipeline of the Low Frequency Array, LOFAR, a leading radio telescope. We tested the tool by running the pipeline on several different compute platforms and collected the performance data. Based on this data, we make well informed recommendations on future hardware and software upgrades. The aim of these upgrades is to accelerate the slowest processing steps for this LOFAR pipeline. The pipeline collector suite is open source and will be incorporated in future LOFAR pipelines to create a performance database for all LOFAR processing.