HARP: Autoregressive Latent Video Prediction with High-Fidelity Image Generator

Abstract

Video prediction is an important yet challenging problem; burdened with the tasks of generating future frames and learning environment dynamics. Recently, autoregressive latent video models have proved to be a powerful video prediction tool, by separating the video prediction into two sub-problems: pre-training an image generator model, followed by learning an autoregressive prediction model in the latent space of the image generator. However, successfully generating high-fidelity and high-resolution videos has yet to be seen. In this work, we investigate how to train an autoregressive latent video prediction model capable of predicting high-fidelity future frames with minimal modification to existing models, and produce high-resolution (256x256) videos. Specifically, we scale up prior models by employing a high-fidelity image generator (VQ-GAN) with a causal transformer model, and introduce additional techniques of top-k sampling and data augmentation to further improve video prediction quality. Despite the simplicity, the proposed method achieves competitive performance to state-of-the-art approaches on standard video prediction benchmarks with fewer parameters, and enables high-resolution video prediction on complex and large-scale datasets.

Multi-View Masked World Models for Visual Robotic Manipulation

“…These setups include (i) a multi-view control where agents operate on multi-view data, (ii) a single-view control where agents operate on single-view data but use auxiliary cameras for representation learning, and (iii) a viewpoint-robust control where agents operate on single randomized viewpoint but use multiple randomized viewpoints for representation learning, as illustrated in Figure 1. Our experiments on RLBench [10] show that our method outperforms single-view representation learning baselines [11,12,13] and a multi-view representation learning baseline [6]. • We show MV-MWM can solve real-world robotic manipulation tasks by transferring a policy trained in simulation to a real-robot without camera calibration.…”

“…We present Multi-View Masked World Models (MV-MWM), a reinforcement learning framework that learns multi-view representations and utilizes them for visual robotic manipulation. Our method builds on top of the Masked World Models (MWM; [13]) framework, which learns a world model on frozen masked autoencoder features. We first introduce how to learn multi-view representations in Section 3.1.…”

“…Our main idea is to train a video masked autoencoder [33,34] with view-masking to reconstruct missing pixels of randomly masked viewpoints. We also incorporate the idea of a prior work [13] that masks convolutional features instead of pixel patches [12] and predicts rewards to learn representations capturing fine-grained details required for visual control. We first describe each component in detail and provide the formal objective.…”

“…Convolutional feature embedding Unlike prior work that masks random pixel patches [12], we embed camera observations into convolutional feature maps and mask these features following the design of Seo et al [13]. This is based on the observation where masked image modeling with pixel patch masking can make it difficult for the model to learn fine-grained details within patches.…”

“…The decoder processes them through ViT layers and linearly projects them into pixel patch predictions. We also follow the idea of Seo et al [13] that predicts a reward to encode task-relevant information.…”