SDC-Net: Video Prediction Using Spatially-Displaced Convolution

Abstract

Fig. 1. Frame prediction on a YouTube video frame featuring a panning camera. Left to right: MCNet [34] result, and our SDC-Net result. The SDC-Net predicted frame is sharper and preserves fine image details, while color distortion and blurriness is seen in the tree and text in MCNet's predicted frame.Abstract. We present an approach for high-resolution video frame prediction by conditioning on both past frames and past optical flows. Previous approaches rely on resampling past frames, guided by a learned future optical flow, or on direct generation of pixels. Resampling based on flow is insufficient because it cannot deal with disocclusions. Generative models currently lead to blurry results. Recent approaches synthesis a pixel by convolving input patches with a predicted kernel. However, their memory requirement increases with kernel size. Here, we present spatially-displaced convolution (SDC) module for video frame prediction. We learn a motion vector and a kernel for each pixel and synthesize a pixel by applying the kernel at a displaced location in the source image, defined by the predicted motion vector. Our approach inherits the merits of both vector-based and kernel-based approaches, while ameliorating their respective disadvantages. We train our model on 428K unlabelled 1080p video game frames. Our approach produces state-of-the-art results, achieving an SSIM score of 0.904 on high-definition YouTube-8M videos, 0.918 on Caltech Pedestrian videos. Our model handles large motion effectively and synthesizes crisp frames with consistent motion.

Improving Semantic Segmentation via Video Propagation and Label Relaxation

“…We hope our approach inspires other ways to perform data augmentation, such as GANs [26], to enable cheap dataset collection and achieve improved accuracy in target tasks. For future work, we would like to explore soft label relaxation using the learned kernels in [34] for better uncertainty reasoning. Our state-of-the-art implementation, will be made publicly available to the research community.…”

“…In our implementation, we use the vector-based architecture as described in [34]. G is a fully convolutional U-net architecture, complete with an encoder and decoder and skip connections between encoder/decoder layers of the same output dimensions.…”

Disentangling Propagation and Generation for Video Prediction

“…On the contrary, motion-based methods [8,9] excel in making sharp predictions, yet fail in occlusion areas where motion predictions are erroneous or ill-defined. Meanwhile, Reda et al [34] propose to model moving appearances with both convolutional kernels as in [10] and vectors as optical flow. Our closest prior work is [11] which also composes the pixel-and flow-based predictions through occlusion maps.…”

Zoom-In-To-Check: Boosting Video Interpolation via Instance-Level Discrimination

“…[28] untangles the memory of the past from the prediction of the future by learning to predict sampling kernels. [22] combines flow-based and kernel-based approaches to learn a model to predict a motion vector and a kernel simultaneously for each pixel.…”