Playable Video Generation

Menapace, Willi; Lathuilière, Stéphane; Tulyakov, Sergey; Siarohin, Aliaksandr; Ricci, Elisa

doi:10.1109/cvpr46437.2021.00993

Cited by 26 publications

(44 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Point-to-point generation [71], a variant of future video prediction, specifies both start and end frames. State or action-conditioned synthesis [26,46] allows to guide the frame-by-frame evolution with high-level commands. Additional works consider video synthesis based on one [49] or multiple layouts [44,70], another video [8], or sound [10,32,69].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

CCVS: Context-aware Controllable Video Synthesis

Moing¹,

Ponce²,

Schmid³

2021

Preprint

View full text Add to dashboard Cite

This presentation introduces a self-supervised learning approach to the synthesis of new video clips from old ones, with several new key elements for improved spatial resolution and realism: It conditions the synthesis process on contextual information for temporal continuity and ancillary information for fine control. The prediction model is doubly autoregressive, in the latent space of an autoencoder for forecasting, and in image space for updating contextual information, which is also used to enforce spatio-temporal consistency through a learnable optical flow module. Adversarial training of the autoencoder in the appearance and temporal domains is used to further improve the realism of its output. A quantizer inserted between the encoder and the transformer in charge of forecasting future frames in latent space (and its inverse inserted between the transformer and the decoder) adds even more flexibility by affording simple mechanisms for handling multimodal ancillary information for controlling the synthesis process (e.g., a few sample frames, an audio track, a trajectory in image space) and taking into account the intrinsically uncertain nature of the future by allowing multiple predictions. Experiments with an implementation of the proposed approach give very good qualitative and quantitative results on multiple tasks and standard benchmarks.Preprint. Under review.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Some sample frames synthesized for these tasks are shown in Figure 4. CCVS creates plausible high-quality videos in various settings, and true interactions with objects compared to previous attempts [46] at the same resolution. Kinetics.…”

Section: Ablation Studymentioning

confidence: 99%

CCVS: Context-aware Controllable Video Synthesis

Moing¹,

Ponce²,

Schmid³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Such methods, however, require dense per-frame annotation of actions or poses at training time, which are costly to obtain and make it challenging to employ such approaches in realistic environments. The task of playable video generation [38], has been introduced to address these limitations. In playable video generation, the discrete action space is discovered in an unsupervised manner and can then be used as a conditioning signal in an auto-regressive probabilistic generative model.…”

Section: Introductionmentioning

confidence: 99%

“…In playable video generation, the discrete action space is discovered in an unsupervised manner and can then be used as a conditioning signal in an auto-regressive probabilistic generative model. While obtaining impressive results, with no intermediate supervision and allowing frame-level control over the generative process, [38] is inherently limited to a single subject and a small set of discrete action controls.…”

Section: Introductionmentioning

confidence: 99%

“…Thus, in this work, we aim to allow richer and more granular control over the generated video content, while similarly requiring no supervision of any kind -i.e., having only raw videos as training input, as in [38]. To do so, we make an observation (inspired by early works in vision [28,34,58]) that a video can be effectively decomposed into foreground / background layers.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Layered Controllable Video Generation

Huang¹,

Jin²,

Yi³

et al. 2021

Preprint

View full text Add to dashboard Cite

We introduce layered controllable video generation, where we, without any supervision, decompose the initial frame of a video into foreground and background layers, with which the user can control the video generation process by simply manipulating the foreground mask. The key challenges are the unsupervised foregroundbackground separation, which is ambiguous, and ability to anticipate user manipulations with access to only raw video sequences. We address these challenges by proposing a two-stage learning procedure. In the first stage, with the rich set of losses and dynamic foreground size prior, we learn how to separate the frame into foreground and background layers and, conditioned on these layers, how to generate the next frame using VQ-VAE generator. In the second stage, we fine-tune this network to anticipate edits to the mask, by fitting (parameterized) control to the mask from future frame. We demonstrate the effectiveness of this learning and the more granular control mechanism, while illustrating state-of-the-art performance on two benchmark datasets. We provide a video abstract as well as some video results on gabrielhuang.github.io/layered controllable video generation.

show abstract

Layered Controllable Video Generation

Huang

Jin

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Playable Video Generation

Cited by 26 publications

References 15 publications

CCVS: Context-aware Controllable Video Synthesis

CCVS: Context-aware Controllable Video Synthesis

Layered Controllable Video Generation

Layered Controllable Video Generation

Contact Info

Product

Resources

About