Temporal Generative Adversarial Nets with Singular Value Clipping

Saito, Masaki; Matsumoto, Eiichi; Saito, Shigeji

doi:10.1109/iccv.2017.308

Cited by 319 publications

(336 citation statements)

References 35 publications

Supporting

Mentioning

333

Contrasting

Order By: Relevance

“…[46] proposed generative adversarial networks for video with a spatial-temporal convolutional architecture that disentangles the scene's foreground from the background. TGAN [47] exploited a 1D temporal generator and a 2D image generator for video generation. The temporal generator takes a single latent variable as input and outputs a set of latent variables, while the image generator transforms these latent variables provided by the temporal generator into video frames.…”

Section: Related Workmentioning

confidence: 99%

Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns

Xie¹,

Zhu

2021

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Video sequences contain rich dynamic patterns, such as dynamic texture patterns that exhibit stationarity in the temporal domain, and action patterns that are non-stationary in either spatial or temporal domain. We show that an energy-based spatial-temporal generative ConvNet can be used to model and synthesize dynamic patterns. The model defines a probability distribution on the video sequence, and the log probability is defined by a spatial-temporal ConvNet that consists of multiple layers of spatial-temporal filters to capture spatial-temporal patterns of different scales. The model can be learned from the training video sequences by an "analysis by synthesis" learning algorithm that iterates the following two steps.Step 1 synthesizes video sequences from the currently learned model.Step 2 then updates the model parameters based on the difference between the synthesized video sequences and the observed training sequences. We show that the learning algorithm can synthesize realistic dynamic patterns. We also show that it is possible to learn the model from incomplete training sequences with either occluded pixels or missing frames, so that model learning and pattern completion can be accomplished simultaneously.

show abstract

Section: Related Workmentioning

confidence: 99%

Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns

Xie¹,

Zhu

2021

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…In [43], [44] the authors address this by directly incorporating the time axis in the input and output. For instance, in [43] the authors propose a temporal generator while Yu et. al [44] propose a sequence generator that learns a stochastic policy.…”

Section: The Proposed Approachmentioning

confidence: 99%

Temporarily-Aware Context Modeling Using Generative Adversarial Networks for Speech Activity Detection

Fernando

Sridharan

McLaren

et al. 2020

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

This paper presents a novel framework for Speech Activity Detection (SAD). Inspired by the recent success of multitask learning approaches in the speech processing domain, we propose a novel joint learning framework for SAD. We utilise generative adversarial networks to automatically learn a loss function for joint prediction of the frame-wise speech/ nonspeech classifications together with the next audio segment. In order to exploit the temporal relationships within the input signal, we propose a temporal discriminator which aims to ensure that the predicted signal is temporally consistent. We evaluate the proposed framework on multiple public benchmarks, including NIST OpenSAT' 17, AMI Meeting and HAVIC, where we demonstrate its capability to outperform state-of-the-art SAD approaches. Furthermore, our cross-database evaluations demonstrate the robustness of the proposed approach across different languages, accents, and acoustic environments.

show abstract

“…Evaluating unconditional video generators. Borrowed from the image generation literature [20], the Inception Score (IS) has become one of the established metrics for quality assessment in videos [19,29,34]. IS incorporates the entropy of the class distributions obtained from a separately trained classifier.…”

Section: Related Workmentioning

confidence: 99%

“…Yet, the established experimental protocol evaluates only on video sequences of a fixed length. Indeed, some previous work [19,34] is even tailored to a pre-defined video length, both at training and at inference time.…”

Section: Related Workmentioning

confidence: 99%

Markov Decision Process for Video Generation

Yushchenko¹,

Araslanov

Roth

2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

View full text Add to dashboard Cite

We identify two pathological cases of temporal inconsistencies in video generation: video freezing and video looping. To better quantify the temporal diversity, we propose a class of complementary metrics that are effective, easy to implement, data agnostic, and interpretable. Further, we observe that current state-of-the-art models are trained on video samples of fixed length thereby inhibiting long-term modeling. To address this, we reformulate the problem of video generation as a Markov Decision Process (MDP). The underlying idea is to represent motion as a stochastic process with an infinite forecast horizon to overcome the fixed length limitation and to mitigate the presence of temporal artifacts. We show that our formulation is easy to integrate into the state-of-the-art MoCoGAN framework. Our experiments on the Human Actions and UCF-101 datasets demonstrate that our MDP-based model is more memory efficient and improves the video quality both in terms of the new and established metrics. * This work was done while VY was at TU Darmstadt.1 Note that the model is still conditioned on the particular training data distribution, hence not truly "unconditional". Still, we adhere to the common terminology used in the literature.subsequence samples for training

show abstract

Temporal Generative Adversarial Nets with Singular Value Clipping

Cited by 319 publications

References 35 publications

Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns

Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns

Temporarily-Aware Context Modeling Using Generative Adversarial Networks for Speech Activity Detection

Markov Decision Process for Video Generation

Contact Info

Product

Resources

About