Selective provenance for datalog programs using top-k queries

Abstract

Highly expressive declarative languages, such as datalog, are now commonly used to model the operational logic of dataintensive applications. The typical complexity of such datalog programs, and the large volume of data that they process, call for result explanation. Results may be explained through the tracking and presentation of data provenance, and here we focus on a detailed form of provenance (howprovenance), defining it as the set of derivation trees of a given fact. While informative, the size of such full provenance information is typically too large and complex (even when compactly represented) to allow displaying it to the user. To this end, we propose a novel top-k query language for querying datalog provenance, supporting selection criteria based on tree patterns and ranking based on the rules and database facts used in derivation. We propose an efficient novel algorithm based on (1) instrumenting the datalog program so that, upon evaluation, it generates only relevant provenance, and (2) efficient top-k (relevant) provenance generation, combined with bottom-up datalog evaluation. The algorithm computes in polynomial data complexity a compact representation of the top-k trees which may be explicitly constructed in linear time with respect to their size. We further experimentally study the algorithm performance, showing its scalability even for complex datalog programs where full provenance tracking is infeasible.

Explaining outputs in modern data analytics

“…We are now able to show how our framework naturally captures existing definitions of provenance for Datalog computations [19,23,29,18]. To the best of our knowledge, this is the only class of iterative computations that have been studied so far in the context of data provenance.…”

“…After the seminal work in [23], [19] was the first to identify that annotationbased approaches explode in size when applied to recursive computations. To provide more concise provenance information, [18] proposed an algorithm for retrieving sub-graphs of the How provenance graph based on user-defined patterns. [29] presents an early effort to add How provenance support in a general-purpose operational Datalog engine (LogicBlox).…”

“…Backward tracing makes relatively few assumptions about the structure of the operators, maintains an amount of state no larger than the total data it has processed, and provides query latencies that depend on the size of the query result. However, the naïve application of backward tracing quickly runs into known issues, even for simple dataflows; backward tracing may identify (i) too many inputs to be helpful [18] and (ii) too few inputs to reproduce the output [26]. We give examples for both cases.…”

Explaining Natural Language query results

“…[14,41,38,17,39,22,21,37,48]) while the field of provenance applications has also been broadly studied (e.g. [26,59,67]). A longstanding challenge in this context is the complexity of provenance expressions, leading to difficulties in presenting them in a user-comprehensible manner.…”

Provenance summaries for answers and non-answers

“…PUG generates explanations for a (missing) result by instrumenting the input query to capture only relevant provenance. This idea has also been employed by SelP [2]. The main difference to our work is that we support negation, but no recursion which SelP supports.…”