Scalable probabilistic databases with factor graphs and MCMC

Wick, Michael; McCallum, Andrew; Miklau, Gerome

doi:10.14778/1920841.1920942

Cited by 55 publications

(45 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our effort is inspired by these approaches, but the goal of this work is to understand the extent to which we can handle these analytics tasks with a single unified architecture. Of these approaches, MCDB [27] and Wick et al [51] are the most related in that they propose a single unified interface for uncertain data based on sampling and graphical models respectively. In contrast, we consider data analytics techniques that are modeled as convex programming problems.…”

mentioning

confidence: 99%

Towards a unified architecture for in-RDBMS analytics

Feng

Kumar

Recht

et al. 2012

Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data

172

181

View full text Add to dashboard Cite

The increasing use of statistical data analysis in enterprise applications has created an arms race among database vendors to offer ever more sophisticated in-database analytics. One challenge in this race is that each new statistical technique must be implemented from scratch in the RDBMS, which leads to a lengthy and complex development process. We argue that the root cause for this overhead is the lack of a unified architecture for in-database analytics. Our main contribution in this work is to take a step towards such a unified architecture. A key benefit of our unified architecture is that performance optimizations for analytics techniques can be studied generically instead of an ad hoc, per-technique fashion. In particular, our technical contributions are theoretical and empirical studies of two key factors that we found impact performance: the order data is stored, and parallelization of computations on a single-node multicore RDBMS. We demonstrate the feasibility of our architecture by integrating several popular analytics techniques into two commercial and one open-source RDBMS. Our architecture requires changes to only a few dozen lines of code to integrate a new statistical technique. We then compare our approach with the native analytics tools offered by the commercial RDBMSes on various analytics tasks, and validate that our approach achieves competitive or higher performance, while still achieving the same quality.

show abstract

mentioning

confidence: 99%

Towards a unified architecture for in-RDBMS analytics

Feng

Kumar

Recht

et al. 2012

Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data

172

181

View full text Add to dashboard Cite

show abstract

“…Nath and Domingos [29] studied how to extend belief propagation on factor graphs with new evidence, but without any modification to the structure of the graph. Wick and McCallum [41] proposed a "query-aware MCMC" method. They designed a proposal scheme so that query variables tend to be sampled more frequently than other variables.…”

Section: Incremental Maintenance Of Statistical Inference and Learningmentioning

confidence: 99%

Incremental knowledge base construction using DeepDive

Shin

Wang

et al. 2016

The VLDB Journal

View full text Add to dashboard Cite

Populating a database with unstructured information is a long-standing problem in industry and research that encompasses problems of extraction, cleaning, and integration. Recent names used for this problem include dealing with dark data and knowledge base construction (KBC). In this work, we describe DeepDive, a system that combines database and machine learning ideas to help develop KBC systems, and we present techniques to make the KBC process more efficient. We observe that the KBC process is iterative, and we develop techniques to incrementally produce inference results for KBC systems. We propose two methods for incremental inference, based respectively on sampling and variational techniques. We also study the tradeoff space of these methods and develop a simple rule-based optimizer. DeepDive includes all of these contributions, and we evaluate Deep-Dive on five KBC systems, showing that it can speed up KBC inference tasks by up to two orders of magnitude with negligible impact on quality.

show abstract

“…Instead, we use a general and flexible representation of a probabilistic database, proposed by Sen et al [36] and also used by Wick et al [42], that can capture complex correlations among the tuples or the attributes in the database through use of factor graphs, a class of graphical models that generalizes both directed Bayesian networks and undirected Markov networks. More formally we have:…”

Section: Pr(w)mentioning

confidence: 99%

“…These include applications such as information extraction, data integration, computer vision applications, sensor networks, etc., where heavy use of machine learning techniques naturally results in complex correlations among the data. Hence, over the last few years, several probabilistic database systems have been proposed that can manage such correlated databases [37,41,42], with correlations typically captured using graphical models such as factor graphs or Bayesian networks [15]. However, intensional approaches that can process queries over such correlated databases and handle more general classes of queries, are typically much slower than extensional approaches and have poor scalability, leading to a significant efficiency gap between the two approaches [21].…”

mentioning

confidence: 99%

“…In this work, we aim to increase the efficiency of intensional methods by developing a framework that represents correlations using factor graphs [37,42], and can exploit context-specific independence and determinism in the correlations, collectively referred to as local structure [13]. Context-specific independence [3,43], often observed in practice, refers to independences that hold given a specific assignment of values to certain variables.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Local structure and determinism in probabilistic databases

Ρεκατσίνας

Deshpande

Getoor

2012

Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data

View full text Add to dashboard Cite

While extensive work has been done on evaluating queries over tuple-independent probabilistic databases, query evaluation over correlated data has received much less attention even though the support for correlations is essential for many natural applications of probabilistic databases, e.g., information extraction, data integration, computer vision, etc. In this paper, we develop a novel approach for efficiently evaluating probabilistic queries over correlated databases where correlations are represented using a factor graph, a class of graphical models widely used for capturing correlations and performing statistical inference. Our approach exploits the specific values of the factor parameters and the determinism in the correlations, collectively called local structure, to reduce the complexity of query evaluation. Our framework is based on arithmetic circuits, factorized representations of probability distributions that can exploit such local structure. Traditionally, arithmetic circuits are generated following a compilation process and can not be updated directly. We introduce a generalization of arithmetic circuits, called annotated arithmetic circuits, and a novel algorithm for updating them, which enables us to answer probabilistic queries efficiently. We present a comprehensive experimental analysis and show speed-ups of at least one order of magnitude in many cases.

show abstract

Scalable probabilistic databases with factor graphs and MCMC

Cited by 55 publications

References 23 publications

Towards a unified architecture for in-RDBMS analytics

Towards a unified architecture for in-RDBMS analytics

Incremental knowledge base construction using DeepDive

Local structure and determinism in probabilistic databases

Contact Info

Product

Resources

About