A Data Lake Prototype Systemfor Multiwavelength (Mwa) Andmultimessenger Astronomy (Mma)

Abstract

At the beginning of the 21st century, large telescopes are coming into operation, making astronomy a Big Data science like few others. Recently, given this scenario, applied computing to astronomy and space sciences has pointed out major challenges involving data science and data engineering. Faced with such a challenge, this article presents a first prototype of a data lake as a strategic solution for heterogeneous data astronomy such as multi-wavelength astronomy (MWA), also including multi-messenger astronomy (MMA). The proposed prototype presents a three-layer architecture, composed of storage, processing and application layers. The storage layer incorporates S3 for raw data storage, MariaDB for simple data storage (crud) and TimescaleDB for complex data storage. The processing layer includes Laravel for CRUD operations and Python for handling large volumes of data. The application layer uses ReactJS with the Cubes.dev library for analytical data visualization. The solution proposes to deliver, when necessary, agility and concatenation between heterogeneous data (light curve, spectra and images) coming from different silos (instruments and telescopes). Thus, the solution, within the MWA and MMA paradigms, incorporates data cube technology and can be customized for applications other than astronomy, such as space weather monitoring and debriefing in the context of space medicine.