A Paralleled Large-Scale Astronomical Cross-Matching Function

Zhao, Qiuyan; Sun, Jizhou; Yu, Ce; Cui, Chenzhou; Lv, Liqiang; Xiao, Jian

doi:10.1007/978-3-642-03095-6_57

Cited by 27 publications

(16 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To increase the speed of cross-matching calculations, multiple technologies have been employed, including high-performing computing (HPC) and sky partitioning. There are acceleration methods based on MPI (Zhao et al 2009), the multi GPU environment (Budavari & Lee 2013), the CPU-GPU cluster (Jia et al 2015), the Hadoop ecological system (Li et al 2014), and Spark framework (?). Additionally, various cross-matching tools have been recently developed to handle massive catalogues, such as ARCHES (Motch et al 2017), C 3 (Riccio et al 2017), and catsHTM (Soumagnac & Ofek 2018).…”

Section: Matching Calculationmentioning

confidence: 99%

“…The use of sky partitioning and parallel processing methods is accompanied by the emergence of boundary problems. The methods to address the problems include increasing redundant data on the boundary (Zhao et al 2009) and using two indexes (Peng et al 2014). However, most of the aforementioned cross-matching works are less appropriate for direct application to time series data reconstruction.…”

Section: Matching Calculationmentioning

confidence: 99%

“…The common method for addressing the above boundary problems is to increase the boundary redundant data by a quick bit-operation algorithm (Zhao et al 2009). However, the implementation of the traditional method is complex and inefficient.…”

Section: Ts-matching Algorithmmentioning

confidence: 99%

See 2 more Smart Citations

AstroCatR: a mechanism and tool for efficient time series reconstruction of large-scale astronomical catalogues

Tang

et al. 2020

Monthly Notices of the Royal Astronomical Society

Self Cite

View full text Add to dashboard Cite

ABSTRACT Time series data of celestial objects are commonly used to study valuable and unexpected objects such as extrasolar planets and supernova in time domain astronomy. Due to the rapid growth of data volume, traditional manual methods are becoming extremely hard and infeasible for continuously analysing accumulated observation data. To meet such demands, we designed and implemented a special tool named AstroCatR that can efficiently and flexibly reconstruct time series data from large-scale astronomical catalogues. AstroCatR can load original catalogue data from Flexible Image Transport System (FITS) files or data bases, match each item to determine which object it belongs to, and finally produce time series data sets. To support the high-performance parallel processing of large-scale data sets, AstroCatR uses the extract-transform-load (ETL) pre-processing module to create sky zone files and balance the workload. The matching module uses the overlapped indexing method and an in-memory reference table to improve accuracy and performance. The output of AstroCatR can be stored in CSV files or be transformed other into formats as needed. Simultaneously, the module-based software architecture ensures the flexibility and scalability of AstroCatR. We evaluated AstroCatR with actual observation data from The three Antarctic Survey Telescopes (AST3). The experiments demonstrate that AstroCatR can efficiently and flexibly reconstruct all time series data by setting relevant parameters and configuration files. Furthermore, the tool is approximately 3× faster than methods using relational data base management systems at matching massive catalogues.

show abstract

Section: Matching Calculationmentioning

confidence: 99%

Section: Matching Calculationmentioning

confidence: 99%

See 1 more Smart Citation

AstroCatR: a mechanism and tool for efficient time series reconstruction of large-scale astronomical catalogues

Tang

et al. 2020

Monthly Notices of the Royal Astronomical Society

Self Cite

View full text Add to dashboard Cite

show abstract

“…Besides, some recent works [11] [12] propose customized solutions to execute some astronomical queries in a parallel architecture. In [11], the authors introduce a cross-matching function including a partitioning approach with HEALPix. OPEN SKYQUERY [12] allows querying and cross-matching distributed astronomical datasets.…”

Section: Related Workmentioning

confidence: 99%

Efficient astronomical query processing using spark

Brahem

Yeh

Zeitouni

2018

Proceedings of the 26th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems

View full text Add to dashboard Cite

Sky surveys represent a fundamental data source in astronomy. Today, these surveys are moving into a petascale regime produced by modern telescopes. Due to the exponential growth of astronomical data, there is a pressing need to provide efficient astronomical query processing. Our goal is to bridge the gap between existing distributed systems and high-level languages for astronomers. In this paper, we present efficient techniques for query processing of astronomical data using ASTROIDE. Our framework helps astronomers to take advantage of the richness of the astronomical data. The proposed model supports complex astronomical operators expressed using ADQL (Astronomical Data Query Language), an extension of SQL commonly used by astronomers. ASTROIDE proposes spatial indexing and partitioning techniques to better filter the data access. It also implements a query optimizer that injects spatial-aware optimization rules and strategies. Experimental evaluation based on real datasets demonstrates that the present framework is scalable and efficient.

show abstract

“…Then astronomers use star catalogs for further analysis. And cross-certification is of crucial importance for data integration [15] . To put it simply, crosscertification means finding out the nearest points on a sphere.…”

Section: B Spatial Applicationsmentioning

confidence: 99%

Accelerating Spatial Data Processing with MapReduce

Wang¹,

Han²,

Tu³

et al. 2010

2010 IEEE 16th International Conference on Parallel and Distributed Systems

View full text Add to dashboard Cite

Abstract-MapReduce is a key-value based programming model and an associated implementation for processing large data sets. It has been adopted in various scenarios and seems promising. However, when spatial computation is expressed straightforward by this key-value based model, difficulties arise due to unfit features and performance degradation. In this paper, we present methods as follows: 1) a splitting method for balancing workload, 2) pending file structure and redundant data partition dealing with relation between spatial objects, 3) a strip-based two-direction plane sweeping algorithm for computation accelerating. Based on these methods, ANN(All nearest neighbors) query and astronomical cross-certification are developed. Performance evaluation shows that the MapReduce-based spatial applications outperform the traditional one on DBMS.

show abstract

A Paralleled Large-Scale Astronomical Cross-Matching Function

Cited by 27 publications

References 1 publication

AstroCatR: a mechanism and tool for efficient time series reconstruction of large-scale astronomical catalogues

AstroCatR: a mechanism and tool for efficient time series reconstruction of large-scale astronomical catalogues

Efficient astronomical query processing using spark

Accelerating Spatial Data Processing with MapReduce

Contact Info

Product

Resources

About