Dynamical Variational Autoencoders: A Comprehensive Review

Abstract

The Variational Autoencoder (VAE) is a powerful deep generative model that is now extensively used to represent high-dimensional complex data via a lowdimensional latent space that is learned in an unsupervised manner. In the original VAE model, input data vectors are processed independently. In the recent years, a series of papers have presented different extensions of the VAE to sequential data, that not only model the latent space, but also model the temporal dependencies within a sequence of data vectors and/or corresponding latent vectors, relying on recurrent neural networks or state space models. In this paper we perform an extensive literature review of these models. Importantly, we introduce and discuss a general class of models called Dynamical Variational Autoencoders (DVAEs) that encompass a large subset of these temporal VAE extensions. Then we present in details seven different instances of DVAE that were recently proposed in the literature, with an effort to homogenize the notations and presentation lines, as well as to relate those models with existing classical temporal models (that are also presented for the sake of completeness). We reimplemented those seven DVAE models and we present the results of an experimental benchmark that we conducted on the speech analysis-resynthesis task (the PyTorch code will be made publicly available). An extensive discussion is presented at the end of the paper, aiming to comment on important issues concerning the DVAE class of models and to describe future research guidelines.

Conditional Generation of Medical Time Series for Extrapolation to Underrepresented Populations

“…The SRNN model is chosen to represent the "classical" dynamic VAE model: it utilizes RNNs as encoder and decoder and models the internal dynamics of the inferred latent sequence with an explicit transition model. In the comprehensive comparison between DVAE models provided by Girin et al [18] it emerges as the most performative model, leading us to select it as the representative for this class of generative models.…”

“…The choices of hyperparameters for each model can be found in Table B.5. For the dynamical VAE baselines (SRNN and KVAE), we largely used the implementations provided by Girin et al [18], with some slight changes and extensions. The implementation of our model, as well as the experimental pipeline necessary to reproduce our results, can be found at https://github.com/simonbing/HealthGen.…”

Introducing Variational Autoencoders to High School Students

“…While these approaches successfully taught students how generative models work through creative applications, they focused on GANs, which are not the only generative AI model. In this work, we focus on VAEs (Kingma and Welling 2013), another powerful generative model extensively used to represent highdimensional data via a low-dimensional latent space (Girin et al 2020). Inspired by previous work, students' learning is guided through playing a simulated "Shadow Matching Game" that teaches students the constituents of a VAE, and an exploration with tools that use VAEs to create media.…”

Physics-guided Deep Markov Models for Learning Nonlinear Dynamical Systems with Uncertainty

“…While RNNs form deterministic models, Dynamic Bayesian Networks, 20,21 accounting for temporal dependencies (such as Hidden Markov Models 22 ), form probabilistic approaches for learning the structure of generative models and enjoy widespread adoption for sequential data. Very recently, Dynamical Variational Autoencoders 23 emerged as a sequential version of variational autoencoders (VAE 24 ), or as a variational version of Dynamical Bayesian Networks, and have been applied to learning the latent space representation for high-dimensional sequential data in an unsupervised manner. These models typically parameterize the involved distributions by means of deep neural networks, which allow for learning high-dimensional and highly multi-modal distributions.…”