Low-cost high performance distributed data storage for multi-channel observations

Abstract

The New Vacuum Solar Telescope (NVST) is a 1-meter solar telescope that aims to observe the fine structures in both the photosphere and the chromosphere of the Sun. The observational data acquired simultaneously from one channel for the chromosphere and two channels for the photosphere bring great challenges to the data storage of NVST. The multi-channel instruments of NVST, including scientific cameras and multi-band spectrometers, generate at least 3 terabytes data per day and require high access performance while storing massive short-exposure images. It is worth studying and implementing a storage system for NVST which would balance the data availability, access performance and the cost of development. In this paper, we build a distributed data storage system (DDSS) for NVST and then deeply evaluate the availability of real-time data storage on a distributed computing environment. The experimental results show that two factors, i.e., the number of concurrent read/write and the file size, are critically important for improving the performance of data access on a distributed environment. Referring to these two factors, three strategies for storing FITS files are presented and implemented to ensure the access performance of the DDSS under conditions of multi-host write and read simultaneously. The real applications of the DDSS proves that the system is capable of meeting the requirements of NVST real-time high performance observational data storage. Our study on the DDSS is the first attempt for modern astronomical telescope systems to store real-time observational data on a low-cost distributed system. The research results and corresponding techniques of the DDSS provide a new option for designing real-time massive astronomical data storage system and will be a reference for future astronomical data storage.

High-performance Negative Database for Massive Data Management System of The Mingantu Spectral Radioheliograph

“…A new data archiving system for the NVST based on the Fastbit NoSQL database has been implemented, where the average number of records generated by NVST in one year is approximately 10 to 12 million (Liu et al 2014). A Distributed Data Storage System (DDSS) for real-time observational data storage based on the Lustre distributed file system was implemented on the NVST (Liu et al 2015).…”

AstroDLLC: Efficiently Reducing Storage and Transmission Costs for Massive Solar Observation Data via Deep Learning-based Lossless Compression

“…Concurrently, the Fivehundred-meter Aperture Spherical Radio Telescope (FAST) is projected to accumulate 7-10 PB of pulsar data each year (Yue & Li 2019). In contrast, the New Vacuum Solar Telescope (NVST) in China, a symbol of cutting-edge optical telescope, is expected to amass at least 3 terabytes (TB) of data daily via the deployment of scientific cameras and multiband spectrometers (Liu et al 2015).…”