Towards compositional probabilistic programming

Abstract

List of AcronymsCK Curse of Knowledge. The tendency to communicate as if others know what we know, because actually thinking about what they will understand takes effort beyond just thinking of what we want to communicate.DSL domain specific language. A programming language designed specifically for a narrow problem domain, often by embedding in a general-purpose language that provides infrastructure. ELBOEvidence Lower Bound. Lower bound on the log-evidence of a generative model. EUBO Evidence Upper Bound. Upper bound on the log-evidence of a generative model. fMRI functional Magnetic Resonance Imaging. Functional magnetic resonance imaging measures brain activity by detecting changes associated with blood flow. GMM Gaussian mixture model. A probabilistic generative model (PGM) of clustering in which cluster indices and parameters are hidden but learned by how they parameterize the observation of continuous data points with a Gaussian likelihood. HMM Hidden Markov Model. A state space model (SSM) in which the latent state has Markovian dynamics, i.e. the state at each time-step depends only on the immediately previous time-step. KL Kullback-Leibler divergence. The average log ratio of two probability densities q and p over the same sample space. MLE maximum likelihood estimation. Calculation of a point estimate of latent variables or model parameters to maximize the likelihood. PDF probability density function. Intuitively, the derivative of a probability measure around a single point in a sample space. vi PGM probabilistic generative model. A statistical model, written in terms of probability densities, that describes a process for generating possible observations.PPL probabilistic programming language. A programming language augmented with primitives for reasoning about uncertainty, usually in the forms of random sampling and observation.SMC symmetric monoidal category. A category with, in addition to the typical sequential composition, a parallel composition whose ordering can be arbitrarily associated or flipped around.SSM state space model. A generative model for time-series data in which each observation is generated based upon the time-evolution of a latent variable through a state space.SPW strictly properly weighted. A relationship between a probability measure and an unnormalized measure whereby drawing samples and weights from the probability measure will, in expectation, equal the value of the unnormalized measure or density. VAE variational autoencoder. A deep generative modeling framework in which not only do neural networks parameterize conditional densities of the generative model, the posterior density is approximated with a neural proposal and the networks are all trained jointly by variational methods.vii Besides my family, I would first and foremost like to express my gratitude to my advisor, Jan-Willem van de Meent, and my co-advisors, Lisa Feldman Barrett, and Karen Quigley. I sent them a cold email late one November, in between tech jobs, while trying to put together graduate applications....