A Compositional Atlas of Tractable Circuit Operations: From Simple Transformations to Complex Information-Theoretic Queries

Abstract

Circuit representations are becoming the lingua franca to express and reason about tractable generative and discriminative models. In this paper, we show how complex inference scenarios for these models that commonly arise in machine learning-from computing the expectations of decision tree ensembles to information-theoretic divergences of deep mixture models-can be represented in terms of tractable modular operations over circuits. Specifically, we characterize the tractability of a vocabulary of simple transformations-sums, products, quotients, powers, logarithms, and exponentials-in terms of sufficient structural constraints of the circuits they operate on, and present novel hardness results for the cases in which these properties are not satisfied. Building on these operations, we derive a unified framework for reasoning about tractable models that generalizes several results in the literature and opens up novel tractable inference scenarios.

Tractable Regularization of Probabilistic Circuits

“…Since the only difference in the structural properties of both PCs classes is determinism, we denote members in the first PC class as deterministic PCs, and members in the second PC class as non-deterministic PCs. Interestingly, both PC classes not only differ in their tractability, which is characterized by the set of queries that can be computed within poly(|p|) time (Vergari et al, 2021), they also exhibit drastically different expressive efficiency. Specifically, abundant empirical (Dang et al, 2020;Peharz et al, 2020a) and theoretical (Choi & Darwiche, 2017) evidences suggest that non-deterministic PCs are more expressive than their deterministic counterparts.…”

“…the model size as well as the size of the original dataset. Entropy regularization The hope for tractable PC entropy regularization comes from the fact that the entropy of a deterministic PC p can be exactly computed in O(|p|) time (Vergari et al, 2021;Shih & Ermon, 2020). However, it is still unclear whether the entropy regularization objective LL ent (θ; D, τ ) (Eq.…”

“…On the other hand, entropy regularization will have different behaviors when applied to non-deterministic PCs. Specifically, since it is coNP-hard to compute the entropy of a non-deterministic PC (Vergari et al, 2021), it is infeasible to optimize the entropy regularization objective LL ent (θ; D, τ ) (Eq. ( 2)).…”

“…PCs are a successful combination of classic probabilistic graphical models (PGMs) and neural networks (NNs). Moreover, by enforcing various structural properties, PCs permit efficient and exact computation of a large family of probabilistic inference queries (Vergari et al, 2021;Khosravi et al, 2019;Shen et al, 2016). The ability to answer these queries leads to successful applications in areas such as model compression and model bias detection (Choi et al, 2020a,b).…”