Exploiting shared correlations in probabilistic databases

Abstract

There has been a recent surge in work in probabilistic databases, propelled in large part by the huge increase in noisy data sourcesfrom sensor data, experimental data, data from uncurated sources, and many others. There is a growing need for database management systems that can efficiently represent and query such data. In this work, we show how data characteristics can be leveraged to make the query evaluation process more efficient. In particular, we exploit what we refer to as shared correlations where the same uncertainties and correlations occur repeatedly in the data. Shared correlations occur mainly due to two reasons: (1) Uncertainty and correlations usually come from general statistics and rarely vary on a tuple-to-tuple basis; (2) The query evaluation procedure itself tends to re-introduce the same correlations. Prior work has shown that the query evaluation problem on probabilistic databases is equivalent to a probabilistic inference problem on an appropriately constructed probabilistic graphical model (PGM). We leverage this by introducing a new data structure, called the random variable elimination graph (rv-elim graph) that can be built from the PGM obtained from query evaluation. We develop techniques based on bisimulation that can be used to compress the rv-elim graph exploiting the presence of shared correlations in the PGM, the compressed rv-elim graph can then be used to run inference. We validate our methods by evaluating them empirically and show that even with a few shared correlations significant speed-ups are possible.

Towards a unified architecture for in-RDBMS analytics

“…Recently, there has been an escalating arms race to add sophisticated analytics into the RDBMS with each iteration bringing more sophisticated tools into the RDBMS. So far, this arms race has centered around bringing individual statistical data mining techniques into an RDBMS, notably Support Vector Machines [35], Monte Carlo sampling [27,51], Conditional Random Fields [25,49], and Graphical Models [43,50]. Our effort is inspired by these approaches, but the goal of this work is to understand the extent to which we can handle these analytics tasks with a single unified architecture.…”

Uncertainty Handling in Named Entity Extraction and Disambiguation for Informal Text

“…The work of [19] represents dependencies resulting from queries with a tree of RV assignments. We are also investigating the shared correlations work of [20].…”

A generic framework for handling uncertain data with local correlations

“…Note that, our model is generic enough to cover the previously proposed model that assumes either object independence [6,25] or global correlations [28,29,31,16]. Specifically, they, respectively, correspond to two special cases: 1) the database has multiple LCPs, each containing only one object, and 2) the database has only one LCP, which contains all objects correlated with each other.…”

“…Furthermore, the probabilistic graphical model (PGM) [14] has been adopted in some works [28,29,31] to represent probabilistic relational table(s) with globally correlated data (i.e., every pair of uncertain objects are assumed to be correlated). These works usually consider relational queries on these tables, including select, project, and join.…”