Wasserstein Introspective Neural Networks

Lee, Kwonjoon; Xu, Weiran; Fan, Fan; Tu, Zhuowen

doi:10.1109/cvpr.2018.00390

Cited by 48 publications

(62 citation statements)

References 25 publications

(62 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Building on the pioneering work of [34], recently [35], [36], [37] have developed an introspective learning method to learn the energy-based model, where the energy function is discriminatively learned, and the learned energy function is used to generate synthesized examples via Langevin dynamics. It can be interesting to combine introspective learning with the proposed cooperative learning method that recruits a generator to jumpstart the Langevin sampling.…”

Section: Related Workmentioning

confidence: 99%

Cooperative Training of Descriptor and Generator Networks

Xie¹,

Gao

et al. 2020

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

This paper studies the cooperative training of two generative models for image modeling and synthesis. Both models are parametrized by convolutional neural networks (ConvNets). The first model is a deep energy-based model, whose energy function is defined by a bottom-up ConvNet, which maps the observed image to the energy. We call it the descriptor network. The second model is a generator network, which is a non-linear version of factor analysis. It is defined by a top-down ConvNet, which maps the latent factors to the observed image. The maximum likelihood learning algorithms of both models involve MCMC sampling such as Langevin dynamics. We observe that the two learning algorithms can be seamlessly interwoven into a cooperative learning algorithm that can train both models simultaneously. Specifically, within each iteration of the cooperative learning algorithm, the generator model generates initial synthesized examples to initialize a finite-step MCMC that samples and trains the energy-based descriptor model. After that, the generator model learns from how the MCMC changes its synthesized examples. That is, the descriptor model teaches the generator model by MCMC, so that the generator model accumulates the MCMC transitions and reproduces them by direct ancestral sampling. We call this scheme MCMC teaching. We show that the cooperative algorithm can learn highly realistic generative models. Index Terms-Deep generative models; Energy-based models; Latent variable models; Bottom-up and top-down convolutional neural networks; Modified contrastive divergence; MCMC teaching !

show abstract

Section: Related Workmentioning

confidence: 99%

Cooperative Training of Descriptor and Generator Networks

Xie¹,

Gao

et al. 2020

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…Our model in (2) directly corresponds to a classifier, specifically a spatial-temporal discriminative ConvNet, in the following sense [22], [51], [52], [53], [54], [55]. Suppose we have C categories of video sequences.…”

Section: Energy-based Spatial-temporal Generative Convnetmentioning

confidence: 99%

Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns

Xie¹,

Zhu

2021

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Video sequences contain rich dynamic patterns, such as dynamic texture patterns that exhibit stationarity in the temporal domain, and action patterns that are non-stationary in either spatial or temporal domain. We show that an energy-based spatial-temporal generative ConvNet can be used to model and synthesize dynamic patterns. The model defines a probability distribution on the video sequence, and the log probability is defined by a spatial-temporal ConvNet that consists of multiple layers of spatial-temporal filters to capture spatial-temporal patterns of different scales. The model can be learned from the training video sequences by an "analysis by synthesis" learning algorithm that iterates the following two steps.Step 1 synthesizes video sequences from the currently learned model.Step 2 then updates the model parameters based on the difference between the synthesized video sequences and the observed training sequences. We show that the learning algorithm can synthesize realistic dynamic patterns. We also show that it is possible to learn the model from incomplete training sequences with either occluded pixels or missing frames, so that model learning and pattern completion can be accomplished simultaneously.

show abstract

“…Our paper is a generalization of [15,56] by applying ABP to train a conditional version of the generator model for feature-to-feature translation. The generative ConvNet [58] and the Wasserstein INN [23] are two one-piece models that learn energy-based generative models for data generation. Both [58] and [23] generate data via iterative MCMC sampling, while our model generates data via direct ancestral sampling, which is much more efficient.…”

Section: Related Workmentioning

confidence: 99%

“…The generative ConvNet [58] and the Wasserstein INN [23] are two one-piece models that learn energy-based generative models for data generation. Both [58] and [23] generate data via iterative MCMC sampling, while our model generates data via direct ancestral sampling, which is much more efficient.…”

Section: Related Workmentioning

confidence: 99%

Learning Feature-to-Feature Translator by Alternating Back-Propagation for Generative Zero-Shot Learning

Zhu

Xie

Liu

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

We investigate learning feature-to-feature translator networks by alternating back-propagation as a generalpurpose solution to zero-shot learning (ZSL) problems. It is a generative model-based ZSL framework. In contrast to models based on generative adversarial networks (GAN) or variational autoencoders (VAE) that require auxiliary networks to assist the training, our model consists of a single conditional generator that maps class-level semantic features and Gaussian white noise vector accounting for instance-level latent factors to visual features, and is trained by maximum likelihood estimation. The training process is a simple yet effective alternating backpropagation process that iterates the following two steps: (i) the inferential back-propagation to infer the latent factors of each observed example, and (ii) the learning backpropagation to update the model parameters. We show that, with slight modifications, our model is capable of learning from incomplete visual features for ZSL. We conduct extensive comparisons with existing generative ZSL methods on five benchmarks, demonstrating the superiority of our method in not only ZSL performance but also convergence speed and computational cost. Specifically, our model outperforms the existing state-of-the-art methods by a remarkable margin up to 3.1% and 4.0% in ZSL and generalized ZSL settings, respectively.

show abstract

Wasserstein Introspective Neural Networks

Cited by 48 publications

References 25 publications

Cooperative Training of Descriptor and Generator Networks

Cooperative Training of Descriptor and Generator Networks

Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns

Learning Feature-to-Feature Translator by Alternating Back-Propagation for Generative Zero-Shot Learning

Contact Info

Product

Resources

About