Stochastic Adversarial Video Prediction

Lee, Alex X.; Zhang, Richard; Ebert, Frederik; Abbeel, Pieter; Finn, Chelsea; Levine, Sergey

doi:10.48550/arxiv.1804.01523

Cited by 123 publications

(263 citation statements)

References 0 publications

Supporting

Mentioning

263

Contrasting

Order By: Relevance

“…In Tab. 1, N ÜWA significantly outperforms 2 181 VideoFlow [18] 3 131 LVT [31] 1 126±3 SAVP [20] 2 116 DVD-GAN-FP [7] 1 110 Video Transformer (S) [44] 1 106±3 TriVD-GAN-FP [23] 1 103 CCVS [25] 1 99±2 Video Transformer (L) [44] 1 94±2…”

Section: Comparison With State-of-the-artmentioning

confidence: 99%

NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion

Wu¹,

Ji²,

Ji³

et al. 2021

Preprint

View full text Add to dashboard Cite

Section: Comparison With State-of-the-artmentioning

confidence: 99%

NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion

Wu¹,

Ji²,

Ji³

et al. 2021

Preprint

View full text Add to dashboard Cite

“…Liang et al [28] defined a dual motion Generative Adversarial Net (GAN). Recently a few approaches have solved the issue of blurriness of predictions multiple steps into the future [29], [30], [31]. Despite the remarkable success, they have their own limitations.…”

Section: Related Workmentioning

confidence: 99%

A Framework for Multisensory Foresight for Embodied Agents

Chen¹,

Hosseini²,

Panetta³

et al. 2021

Preprint

View full text Add to dashboard Cite

Predicting future sensory states is crucial for learning agents such as robots, drones, and autonomous vehicles. In this paper, we couple multiple sensory modalities with exploratory actions and propose a predictive neural network architecture to address this problem. Most existing approaches rely on large, manually annotated datasets, or only use visual data as a single modality. In contrast, the unsupervised method presented here uses multi-modal perceptions for predicting future visual frames. As a result, the proposed model is more comprehensive and can better capture the spatio-temporal dynamics of the environment, leading to more accurate visual frame prediction. The other novelty of our framework is the use of sub-networks dedicated to anticipating future haptic, audio, and tactile signals. The framework was tested and validated with a dataset containing 4 sensory modalities (vision, haptic, audio, and tactile) on a humanoid robot performing 9 behaviors multiple times on a large set of objects. While the visual information is the dominant modality, utilizing the additional non-visual modalities improves the accuracy of predictions.

show abstract

“…Video generation is a challenging problem with a lot of applications in robotics [18,33], augmented reality [64,65], data augmentation [4,22,50,74], and action imitation [1,13,37,39,55,60,62]. It has different variations, such as video prediction [28,31,71], video synthesis [64,65], video interpolation [42,54], and super-resolution [14,27,38].…”

Section: Introductionmentioning

confidence: 99%

LARNet: Latent Action Representation for Human Action Synthesis

Biyani¹,

Rana²,

Vyas³

et al. 2021

Preprint

View full text Add to dashboard Cite

We present LARNet, a novel end-to-end approach for generating human action videos. A joint generative modeling of appearance and dynamics to synthesize a video is very challenging and therefore recent works in video synthesis have proposed to decompose these two factors. However, these methods require a driving video to model the video dynamics. In this work, we propose a generative approach instead, which explicitly learns action dynamics in latent space avoiding the need of a driving video during inference. The generated action dynamics is integrated with the appearance using a recurrent hierarchical structure which induces motion at different scales to focus on both coarse as well as fine level action details. In addition, we propose a novel mix-adversarial loss function which aims at improving the temporal coherency of synthesized videos. We evaluate the proposed approach on four real-world human action datasets demonstrating the effectiveness of the proposed approach in generating human actions. Code available at https://github.com/aayushjr/larnet.

show abstract

Stochastic Adversarial Video Prediction

Cited by 123 publications

References 0 publications

NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion

NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion

A Framework for Multisensory Foresight for Embodied Agents

LARNet: Latent Action Representation for Human Action Synthesis

Contact Info

Product

Resources

About