Local structure and determinism in probabilistic databases

Abstract

While extensive work has been done on evaluating queries over tuple-independent probabilistic databases, query evaluation over correlated data has received much less attention even though the support for correlations is essential for many natural applications of probabilistic databases, e.g., information extraction, data integration, computer vision, etc. In this paper, we develop a novel approach for efficiently evaluating probabilistic queries over correlated databases where correlations are represented using a factor graph, a class of graphical models widely used for capturing correlations and performing statistical inference. Our approach exploits the specific values of the factor parameters and the determinism in the correlations, collectively called local structure, to reduce the complexity of query evaluation. Our framework is based on arithmetic circuits, factorized representations of probability distributions that can exploit such local structure. Traditionally, arithmetic circuits are generated following a compilation process and can not be updated directly. We introduce a generalization of arithmetic circuits, called annotated arithmetic circuits, and a novel algorithm for updating them, which enables us to answer probabilistic queries efficiently. We present a comprehensive experimental analysis and show speed-ups of at least one order of magnitude in many cases.

ReStore - Neural Data Completion for Relational Databases

“…The goal is to estimate possible results for queries. Alternatively, uncertainty can be modeled using probabilistic databases [10,18,24,29,31,32,35] where tuples or sets of tuples are annotated with probabilities. In contrast to our work, missing tuples cannot be handled directly.…”

DeepDB

“…Probabilistic Databases. Similar to our work, probabilistic databases have used graphical models to represent joint probability distributions [47,15,36,37] to overcome the tuple-independence assumption that early approaches relied on [6,31,41]. For instance, Markov Logic Networks (MLNs) were used to explicitly specify correlations in probabilistic databases [15,13].…”

“…In contrast to DeepDB , correlations need to be manually specified and are not learned. Moreover, Rekatsinas et al [36] instead use factor graphs to model correlations in probabilistic databases and construct annotated arithmetic circuits (AACs) which also encode a probability distribution with sum and product nodes similar to SPNs. Different from DeepDB , additional representations (lineage-AACs) have to be constructed on a per-query basis whereas RSPNs are data-driven and thus workload-independent.…”

Tree Decomposition-based Approach for Compiling Independent Sets

“…Knowledge compilation has been intensively studied and has applications in various fields. In Broeck et al's study [2], a variety of exemplary applications were cited, such as diagnosis [3], [4], planning [5], probabilistic reasoning [6], [7], [8], probabilistic databases [9], [10], [11], [12], first-order probabilistic inference [13], [14], [15], and learning of tractable probabilistic models [16].…”