Predicting Video with VQVAE

Walker, Jacob; Razavi, Ali; Oord, Aäron van den

doi:10.48550/arxiv.2103.01950

Cited by 16 publications

(20 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A clean architecture with VQ-VAE for video generation has not been presented yet and we hope VideoGPT is useful from that standpoint. While VQ-VAE-2 (Razavi et al, 2019) proposes using multi-scale hierarchical latents and SNAIL blocks (Chen et al, 2017) (and this setup has been applied to videos in (Walker et al, 2021)), the pipeline is inherently complicated and hard to reproduce. For simplicity, ease of reproduction and presenting the first VQ-VAE based video generation model with minimal complexity, we stick with a single scale of discrete latents and transformers for the autoregressive priors, a design choice also adopted in DALL-E (Ramesh et al, 2021).…”

Section: Related Workmentioning

confidence: 99%

VideoGPT: Video Generation using VQ-VAE and Transformers

Yan,

Zhang,

Abbeel

et al. 2021

Preprint

View full text Add to dashboard Cite

We present VideoGPT: a conceptually simple architecture for scaling likelihood based generative modeling to natural videos. VideoGPT uses VQ-VAE that learns downsampled discrete latent representations of a raw video by employing 3D convolutions and axial self-attention. A simple GPTlike architecture is then used to autoregressively model the discrete latents using spatio-temporal position encodings. Despite the simplicity in formulation and ease of training, our architecture is able to generate samples competitive with stateof-the-art GAN models for video generation on the BAIR Robot dataset, and generate high fidelity natural images from UCF-101 and Tumbler GIF Dataset (TGIF). We hope our proposed architecture serves as a reproducible reference for a minimalistic implementation of transformer based video generation models. Samples and code are available at https://wilson1yan. github.io/videogpt/index.html.

show abstract

Section: Related Workmentioning

confidence: 99%

VideoGPT: Video Generation using VQ-VAE and Transformers

Yan,

Zhang,

Abbeel

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…This may, however, be practical for real-world web services and other applications, where users already continually interact with the system and A/B testing is standard practice. End-to-end training could also enable PICO to be applied to problems other than compression, such as image captioning for visually-impaired users, or audio visualization for hearing-impaired users [42] -such applications could also be enabled through continued improvements to generative models for video [43,44], audio [45], and text [46,47]. Another exciting area for future work is to apply pragmatic compression to a wider range of realistic applications, including video compression for robotic space exploration [13], audio compression for hearing aids [48,49], and spatial compression for virtual reality [50].…”

Section: Discussionmentioning

confidence: 99%

Pragmatic Image Compression for Human-in-the-Loop Decision-Making

Reddy,

Dragan,

Levine

2021

Preprint

View full text Add to dashboard Cite

Standard lossy image compression algorithms aim to preserve an image's appearance, while minimizing the number of bits needed to transmit it. However, the amount of information actually needed by a user for downstream tasks -e.g., deciding which product to click on in a shopping website -is likely much lower. To achieve this lower bitrate, we would ideally only transmit the visual features that drive user behavior, while discarding details irrelevant to the user's decisions. We approach this problem by training a compression model through human-in-the-loop learning as the user performs tasks with the compressed images. The key insight is to train the model to produce a compressed image that induces the user to take the same action that they would have taken had they seen the original image. To approximate the loss function for this model, we train a discriminator that tries to distinguish whether a user's action was taken in response to the compressed image or the original. We evaluate our method through experiments with human participants on four tasks: reading handwritten digits, verifying photos of faces, browsing an online shopping catalogue, and playing a car racing video game. The results show that our method learns to match the user's actions with and without compression at lower bitrates than baseline methods, and adapts the compression model to the user's behavior: it preserves the digit number and randomizes handwriting style in the digit reading task, preserves hats and eyeglasses while randomizing faces in the photo verification task, preserves the perceived price of an item while randomizing its color and background in the online shopping task, and preserves upcoming bends in the road in the car racing game.Preprint. Under review.

show abstract

“…Early approaches for this problem typically employed recurrent convolutional models trained with reconstruction objective [17,54,65], but later adversarial losses were introduced to improve the synthesis quality [37,71,75]. Some recent works explore autoregressive video prediction with recurrent or attention-based models (e.g., [27,53,73,78,81]). Another close line of research is video interpolation, i.e.…”

Section: Related Workmentioning

confidence: 99%

StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2

Skorokhodov¹,

Tulyakov²,

Elhoseiny³

2021

Preprint

View full text Add to dashboard Cite

Videos show continuous events, yet most -if not allvideo synthesis frameworks treat them discretely in time.In this work, we think of videos of what they should betime-continuous signals, and extend the paradigm of neural representations to build a continuous-time video generator. For this, we first design continuous motion representations through the lens of positional embeddings. Then, we explore the question of training on very sparse videos and demonstrate that a good generator can be learned by using as few as 2 frames per clip. After that, we rethink the traditional image and video discriminators pair and propose to use a single hypernetwork-based one. This decreases the training cost and provides richer learning signal to the generator, making it possible to train directly on 1024 2 videos for the first time. We build our model on top of StyleGAN2 and it is just ≈5% more expensive to train at the same resolution while achieving almost the same image quality. Moreover, our latent space features similar properties, enabling spatial manipulations that our method can propagate in time. We can generate arbitrarily long videos at arbitrary high frame rate, while prior work struggles to generate even 64 frames at a fixed rate. Our model achieves state-of-the-art results on four modern 256 2 video synthesis benchmarks and one 1024 2 resolution one. 1

show abstract

Predicting Video with VQVAE

Cited by 16 publications

References 29 publications

VideoGPT: Video Generation using VQ-VAE and Transformers

VideoGPT: Video Generation using VQ-VAE and Transformers

Pragmatic Image Compression for Human-in-the-Loop Decision-Making

StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2

Contact Info

Product

Resources

About