Diagnosing and Fixing Manifold Overfitting in Deep Generative Models

Abstract

Likelihood-based, or explicit, deep generative models use neural networks to construct flexible high-dimensional densities. This formulation directly contradicts the manifold hypothesis, which states that observed data lies on a low-dimensional manifold embedded in high-dimensional ambient space. In this paper we investigate the pathologies of maximum-likelihood training in the presence of this dimensionality mismatch. We formally prove that degenerate optima are achieved wherein the manifold itself is learned but not the distribution on it, a phenomenon we call manifold overfitting. We propose a class of two-step procedures consisting of a dimensionality reduction step followed by maximum-likelihood density estimation, and prove that they recover the data-generating distribution in the nonparametric regime, thus avoiding manifold overfitting. We also show that these procedures enable density estimation on the manifolds learned by implicit models, such as generative adversarial networks, hence addressing a major shortcoming of these models. Several recently proposed methods are instances of our two-step procedures; we thus unify, extend, and theoretically justify a large class of models.

Multi-Modal Latent Diffusion

“…To obtain such samples, we follow the two-stage procedure described in [39][40][41], where samples from the lower-dimensional q ψ (z) are obtained through a score-based generative model. These models have shown tremendous performance in fitting complex distributions [10,42], an ability which aligns with our objective of learning the distribution within a multimodal latent space.…”

“…Joint generation: To generate a new sample for all modalities, we use a simple scorebased diffusion model in latent space [32,39,40,42,43]. This requires reversing a stochastic noising process, starting from a simple Gaussian distribution.…”

Closing the gap: Exact maximum likelihood training of generative autoencoders using invertible layers

“…Similarly, [76] and [5] use a deterministic autoencoder to learn the latent representation of the data, and a normalizing flow to model the distribution of such latents, leading to better density estimation while avoiding over-regularization of the latent variables. More recently, [46] discusses the problem of manifold overfitting, which arises when the manifold is learned but not the distribution on it, and propose a two-step training procedure applicable to all the likelihood-based models. NFs on manifolds: Several authors propose variations of NFs that can model data on a manifold.…”