Niffler: A Reference Architecture and System Implementation for View Discovery over Pathless Table Collections by Example

Abstract

Identifying a project-join view (PJ-view) over collections of tables is the first step of many data management projects, e.g., assembling a dataset to feed into a business intelligence tool, creating a training dataset to fit a machine learning model, and more. When the table collections are large and lack join information-such as when combining databases, or on data lakes-query by example (QBE) systems can help identify relevant data, but they are designed under the assumption that join information is available in the schema, and do not perform well on pathless table collections that do not have join path information.We present a reference architecture that explicitly divides the end-to-end problem of discovering PJ-views over pathless table collections into a human and a technical problem. We then present Niffler, a system built to address the technical problem. We introduce algorithms for the main components of Niffler, including a signal generation component that helps reduce the size of the candidate views that may be large due to errors and ambiguity in both the data and input queries. We evaluate Niffler on real datasets to demonstrate the effectiveness of the new engine in discovering PJ-views over pathless table collections.

Join Path-Based Data Augmentation for Decision Trees

“…Using this graph, Aurum outputs all the transitively connected join paths of hop = k given a source and a target node. Niffler [11] uses the same EKG to find all possible combinations of columns, generates only 2-hop join paths between these columns and ranks the paths according tot the score from EKG. Our ranking function is customised for ML classification.…”