Leveraging data lineage to infer logical relationships between astronomical catalogs

Abstract

A novel method to infer logical relationships between sets is presented. These sets can be any collection of elements, for example astronomical catalogs of celestial objects. The method does not require the contents of the sets to be known explicitly. It combines incomplete knowledge about the relationships between sets to infer a priori unknown relationships. Relationships between sets are represented by sets of Boolean hypercubes. This leads to deductive reasoning by application of logical operators to these sets of hypercubes. A pseudo code for an efficient implementation is described. The method is used in the Astro-WISE information system to infer relationships between catalogs of astronomical objects. These catalogs can be very large and, more importantly, their contents do not have to be available at all times. Science products are stored in Astro-WISE with references to other science products from which they are derived, or their dependencies. This creates full data lineage that links every science product all the way back to the raw data. Catalogs are created in a way that maximizes knowledge about their relationship with their dependencies. The presented algorithm is used to determine which objects a catalog represents by leveraging this information.

Automatic optimized discovery, creation and processing of astronomical catalogs

“…A Select Attributes that selects no attributes is placed at the end of the dependency graph, which is subsequently moved up in figure (b). One of the parents of the Concatenate Attributes can be removed, because the algorithm from Buddelmeijer et al (2011a) is used to infer that it is not required anymore. The Concatenate Attributes itself is removed because it has only one parent.…”