Automatic optimized discovery, creation and processing of astronomical catalogs

Abstract

We present the design of a novel way of handling astronomical catalogs in Astro-WISE in order to achieve the scalability required for the data produced by large scale surveys. A high level of automation and abstraction is achieved in order to facilitate interoperation with visualization software for interactive exploration. At the same time flexibility in processing is enhanced and data is shared implicitly between scientists. This is accomplished by using a data model that primarily stores how catalogs are derived; the contents of the catalogs are only created when necessary and stored only when beneficial for performance. Discovery of existing catalogs and creation of new catalogs is done through the same process by directly requesting the final set of sources (astronomical objects) and attributes (physical properties) that is required, for example from within visualization software.New catalogs are automatically created to provide attributes of sources for which no suitable existing catalogs can be found. These catalogs are defined to contain the new attributes on the largest set of sources the calculation of the attributes is applicable to, facilitating reuse for future data requests. Subsequently, only those parts of the catalogs that are required for the requested end product are actually processed, ensuring scalability.The presented mechanisms primarily determine which catalogs are created and what data has to be processed and stored: the actual processing and storage itself is left to existing functionality of the underlying information system.

Leveraging data lineage to infer logical relationships between astronomical catalogs

“…This is taken to the extreme for catalog instances: catalogs can be created and stored without fully processing them, or without processing them at all [1]. In other words, it is not required to create or store the contents of a catalog as a whole, achieving the scalability required to handle large catalogs.…”

“…The method is designed for the Astro-WISE information system to infer relationships between astronomical catalogs [1]. Catalog handling using this method is discussed in the following sections.…”

Query driven visualization of astronomical catalogs

“…Both default administrative messages and widely accepted application defined messages can be found on the SAMP wiki. 3 In the rest of this section we first describe relevant existing messages and subsequently introduce our proposed messages. We list the type of the messages and a description of the intended action of the receiver.…”

“…(2) We permit requests in a more declarative form than direct database queries would allow. (3) We allow the user to inspect and influence the processing from within the visualization by exporting the data lineage. (4) We achieve a high level of abstraction that allows close interoperation between software.…”

“…The creation as well as the disclosure of the catalogs can be handled by data pulling mechanisms (Buddelmeijer et al 2011). These prevent unnecessary processing and facilitate data sharing by having users request the desired end products.…”