Symbolic conditioning of arrays in probabilistic programs

Abstract

Probabilistic programming systems make machine learning more modular by automating inference. Recent work by Shan and Ramsey makes inference more modular by automating conditioning. Their technique uses a symbolic program transformation that treats conditioning generally via the measure-theoretic notion of disintegration. This technique, however, is limited to conditioning a single scalar variable. As a step towards modular inference for realistic machine learning applications, we have extended the disintegration algorithm to symbolically condition arrays in probabilistic programs. The extended algorithm implements lifted disintegration, where repetition is treated symbolically and without unrolling loops. The technique uses a language of index variables for tracking expressions at various array levels. We find that the method works well for arbitrarily-sized arrays of independent random choices, with the conditioning step taking time linear in the number of indices needed to select an element.

Symbolic Disintegration with a Variety of Base Measures

“…Our base-measure language (Figure 22) includes discrete-continuous mixtures over R, dependent products, and disjoint sums; we also allow specifying the base measure as another probabilistic program (Section 7). Thus, our work subsumes all but Narayanan and Shan's [2017] and Roberts et al's [2019] handling of arrays, and is the first to allow different base measures over the same type.…”

“…Shan and Ramsey [2017] showed that disintegration is also a useful probabilistic-program transformation, but only automated it for base measures that are independent products of Lebesgue and counting measures. Narayanan and Shan [2017] generalized those independent products to handle arrays without unrolling. Any disintegration program transformation can also be used to find densities, because the totaling map .…”

“…In fact, density calculation and conditioning are special cases of disintegration. Recent years have seen the exact automation of both density calculation [Bhat et al 2012[Bhat et al , 2013Mohammed Ismail and Shan 2016;Cusumano-Towner et al 2019;Roberts et al 2019] and disintegration [Shan and Ramsey 2017;Narayanan and Shan 2017]. Unlike other implementations of inference and sampling [Lunn et al 2000;Goodman et al 2008;Pfeffer 2009;Wingate et al 2011;Wood et al 2014;Carpenter et al 2017;Wu et al 2018], these mechanizations enable the sampling user to specify a proposal distribution as a probabilistic program.…”

“…The state of the art in mechanizing density calculation on probabilistic programs [Bhat et al 2012[Bhat et al , 2013Mohammed Ismail and Shan 2016;Shan and Ramsey 2017;Narayanan and Shan 2017;Cusumano-Towner et al 2019;Roberts et al 2019] is limited to the stock base measure. Thus, our disintegrator is the first transformation that can turn in (13) into in (15) with respect to in (14), a mixture of the Lebesgue measure and point masses (B1).…”

Calculating Distributions

“…When an observed quantity is not drawn directly from a primitive distribution, but rather a result computed by the model (such as the center of mass of random particles [1]), it is more involved to specify and implement the disintegration program transformation turning (16) into (18) [13,17]. In particular, we recently extended automatic disintegration to applications where the distribution of the observation has no density with respect to the Lebesgue base measure.…”