Trace types and denotational semantics for sound programmable inference in probabilistic languages

Abstract

Modern probabilistic programming languages aim to formalize and automate key aspects of probabilistic modeling and inference. Many languages provide constructs for programmable inference that enable developers to improve inference speed and accuracy by tailoring an algorithm for use with a particular model or dataset. Unfortunately, it is easy to use these constructs to write unsound programs that appear to run correctly but produce incorrect results. To address this problem, we present a denotational semantics for programmable inference in higher-order probabilistic programming languages, along with a type system that ensures that well-typed inference programs are sound by construction. A central insight is that the type of a probabilistic expression can track the space of its possible execution traces, not just the type of value that it returns, as these traces are often the objects that inference algorithms manipulate. We use our semantics and type system to establish soundness properties of custom inference programs that use constructs for variational, sequential Monte Carlo, importance sampling, and Markov chain Monte Carlo inference. CCS Concepts: • Mathematics of computing → Probabilistic inference problems; Variational methods; Metropolis-Hastings algorithm; Sequential Monte Carlo methods; • Theory of computation → Semantics and reasoning; Denotational semantics; • Software and its engineering → Formal language definitions.

Probabilistic programming semantics for name generation

“…In addition to proving full abstraction of the QBS semantics of the -calculus at first order, we provide the first detailed investigation of the higher-typed function spaces in Borel-based probability theory (ğ4.1, ğ5). The application of higher-order probabilistic methods is increasingly widespread in programming research (ğ6.3 and [Ehrhard et al 2018;Lew et al 2019;Sato et al 2019;Ścibior et al 2017;Vandenbroucke and Schrijvers 2020]). We show that our programming-based development can alternatively be viewed in terms of recent categorical formulations of probability theory (ğ5).…”

“…Quasi-Borel spaces ] are a convenient setting including both measure theory and higher-typed function spaces that are increasingly widely used (e.g. [Lew et al 2019;Sato et al 2019;Ścibior et al 2017;Vandenbroucke and Schrijvers 2020]). They work by first restricting probability theory to the well-behaved domain of standard Borel spaces (ğ3.1).…”

Raising expectations: automating expected cost analysis with types

“…Fuzz [Reed and Pierce 2010] uses linear types augmented with a probability monad to reason about differential privacy of randomized computation, and DFuzz [Gaboardi et al 2013] later generalizes it with indexed types and lightweight dependent types to certify differential privacy for a broader class of benchmarks. Recently, Lew et al [Lew et al 2020] have developed a type system for programmable probabilistic inference with trace types, where well-typed inference programs soundly derive posterior distributions by construction. In this paper, we focus on expected cost bound analysis for probabilistic programs.…”

Continualization of Probabilistic Programs With Correction

“…frameworks and systems have been developed for Probabilistic Programming [57,33,63,32,22]. Additionally these analyses make use of a rich set of semantics [44,36,7,64,19], however of particular note is recent work by Lew et al [41], which provides a type system for reasoning about variational approximations; however they focus on continuous approximations of already continuous variables.…”