mcatCS: A Highly Efficient Cross-matching Scheme for Multi-band Astronomical Catalogs

Abstract

Multi-band astronomical catalog cross-matching has always been, and will continue to be, indispensable to astronomy research. However, the archived data volume in different wavebands is extremely huge, which results in the cross-matching process having high computational consumption and slow response. The complexity will also be augmented by the continuous growth of observational data. In this paper, we present mcatCS (multi-band catalog Cross-matching Scheme), a distributed cross-matching scheme to efficiently integrate celestial object data from billion-row multi-band astronomical catalogs. It is deployed on a cluster of commodity machines and provides a command-line-based interface to the end user. To allow fast cross-matching, the data in catalogs are reformatted into the Grouped Spatial Index File, which is a specially designed multi-band catalog uniform format. Furthermore, a min-conflicts data layout strategy is utilized to maximize the parallelization of cross-matching. Using real data, archived in the National Astronomical Observatories of China, we verify that mcatCS has good capabilities for performing efficient and reliable cross-matching between billion-row multi-band catalogs, and experimental results show that the query response speed is 38% to 45% greater than that of MongoDB and 21% to 32% greater than that of PostgreSQL with the HEALPix B-tree index. Moreover, although Q3C and H3C-the extension index packages for PostgreSQL-offer faster query response speed for less than 85 million sources, mcatCS proves to be advantageous after sources scale up to 100 million, and achieves a time reduction of 30.3% and 30.7% compared to Q3C and H3C for 200 million sources.

HLC2: a highly efficient cross-matching framework for large astronomical catalogues on heterogeneous computing environments

“…Compared with the parallel algorithm of declination single dimensional index, the performance of grid indexing is improved by about 60 times. More recently, Li et al ( 2019 ) proposed a multiband cross-matching schema with a specially designed catalogue format and data layout based on KD-tree, which can support faster query response than Q3C and H3C after sources scale up to 100 million. Tree structure has shown considerable usefulness on the retrieval problems.…”

“…explored cross-matching strategies on CPU-GPU clusters. In addition, mcatCS (Li et al 2019 ) is also a distributed cross-matching scheme to efficiently integrate celestial object data. Ho we ver, it focuses on single record cross-matching with low complexity and is therefore timeconsuming for large-scale catalogue matching.…”

A Cross-matching Service for Data Center of Xinjiang Astronomical Observatory

“…Ma et al (2018) proposed E-Zone algorithm, which uses Euclidean distance for faster calculation of adjacent points, and implements parallel calculation based on OpenMP. Li et al (2019) designed a multi-band catalog unified format, combined with the data layout strategy of minimum conflict to improve the parallelization of cross-matching, and achieved 30.3% and 30.7% time reduction compared with Quad Tree Cube (Q3C) and HealpiX-tree-C (H3C) at 200 million data sources of astronomical catalogs. Zhang et al (2023b) proposed a large-scale cross-matching framework supporting heterogeneous computing, which reduced the cross-matching time to 5 s for small-scale astronomical catalogs, 150 s for medium-scale astronomical catalogs, and 260 s for large-scale astronomical catalogs.…”

3DT-CM: A Low-complexity Cross-matching Algorithm for Large Astronomical Catalogues Using 3d-tree Approach

“…To accelerate the cross-matching process in astronomical data fusion, various methods have been employed to reduce computation time, including data division (Gray et al 2007) (Gorski et al 2005) (Szalay et al 2007) and parallel computing (Zečević et al 2019) (Li et al 2019) (Zhao et al 2009). Index partitioning methods have been particularly useful, but they suffer from source leaking at the border of each area.…”