Scalable Data Mining, Archiving, and Big Data Management for the Next Generation Astronomical Telescopes

Abstract

Big data as a paradigm focuses on data volume, velocity, and on the number and complexity of various data formats and metadata, a set of information that describes other data types. This is nowhere better seen than in the development of the software to support next generation astronomical instruments including the MeerKAT/KAT-7 Square Kilometre Array (SKA) precursor in South Africa, in the Low Frequency Array (LOFAR) in Europe, in two instruments led in part by the U.S. National Radio Astronomy Observatory (NRAO) with its Expanded Very Large Array (EVLA) in Socorro, NM, and Atacama Large Millimeter Array (ALMA) in Chile, and in other instruments such as the Large Synoptic Survey Telescope (LSST) to be built in northern Chile. This chapter highlights the big data challenges in constructing data management systems for these astronomical instruments, specifically the challenge of integrating legacy science codes, handling data movement and triage, building flexible science data portals and user interfaces, allowing for flexible technology deployment scenarios, and in automatically and rapidly mitigating the difference in science data formats and metadata models. The authors discuss these challenges and then suggest open source solutions to them based on software from the Apache Software Foundation including Apache Object-Oriented Data Technology (OODT), Tika, and Solr. The authors have leveraged these solutions to effectively and expeditiously build many precursor and operational software systems to handle data from these astronomical instruments and to prepare for the coming data deluge from those not constructed yet. Their solutions are not specific to the astronomical domain and they are already applicable to a number of science domains including Earth, planetary, and biomedicine.

Business-making supported via the application of big data to achieve economic sustainability

“…Full utilization of Big Data depends on effective management, which makes data extraction easier (Wu et al, 2018). Three main issues related to Big Data include: quantity of collected data, speed required for data analysis, and various data formats that are collected (Mattmann et al, 2014;Favaretto et al, 2020).…”

A Machine Learning Classifier for Fast Radio Burst Detection at the VLBA

“…We employ the Catalog and Archive Service (CAS) Crawling Framework (Mattmann et al 2013) to run at predefined intervals and automatically detect new candidate products, which triggers the extraction and storage of the metadata in Solr using the OODT File Manager (Mattmann et al 2013). This metadata includes information about the associated VLBA job's observing parameters (which frequencies and antennas were used and where the array was pointed), the date and time at which the candidate began, the duration of the candidate, the estimated dispersion measure, and more.…”

Revisiting the Anatomy and Physiology of the Grid

“…Today, grids have been used successfully in several domains, including cancer research [3], planetary science [4], earth science [5], and astrophysics [6,40,44].…”