Wide and Narrow: Video Prediction from Context and Motion

Abstract

Video prediction, forecasting the future frames from a sequence of input frames, is a challenging task since the view changes are influenced by various factors, such as the global context surrounding the scene and local motion dynamics. In this paper, we propose a new framework to integrate these complementary attributes to predict complex pixel dynamics through deep networks. To capture the local motion pattern of objects, we devise local filter memory networks that generate adaptive filter kernels by storing the prototypical motion of moving objects in the memory. We further present global context propagation networks that iteratively aggregate the non-local neighboring representations to preserve the contextual information over the past frames. The proposed framework, utilizing the outputs from both networks, can address blurry predictions and color distortion. We conduct experiments on Caltech pedestrian and UCF101 datasets, and demonstrate state-of-the-art results. Especially for multi-step prediction, we obtain an outstanding performance in quantitative and qualitative evaluation.

Prediction of Social Dynamic Agents and Long-Tailed Learning Challenges: A Survey

“…Input Encoding Methods in video prediction typically consist of CNNs with a Unet (Ronneberger et al, 2015) type architecture (e.g., Cho et al, 2021;Ho et al, 2019;Bhattacharjee & Das, 2019;Kwon & Park, 2019;Ying et al, 2019). CNNs are an obvious choice for encoding the sequences of frames within a video due to their effectiveness on image-based data (Russakovsky et al, 2015), and the U-net architecture is an especially versatile form of CNN which can be applied to any application where the 'labels' are of the same format (e.g.…”

“…an image with the same height and width) as the inputs (Ronneberger et al, 2015). Since this is the case for video prediction, U-net CNNs can either be used straightforwardly for encoding purposes (e.g., Cho et al, 2021;Ho et al, 2019) or as the generator within a GAN framework (Isola et al, 2018) to perform adversarial learning (e.g., Bhattacharjee & Das, 2019;Kwon & Park, 2019;Ying et al, 2019).…”

“…Modeling Methods. While most of the reviewed methods use generative adversarial networks (GANs) (e.g., Bhattacharjee & Das, 2019;Kwon & Park, 2019;Ying et al, 2019;Cai et al, 2018;Liang et al, 2017), or variations of autoencoders (e.g., Cho et al, 2021;Oliu et al, 2018;Liu et al, 2018), Byeon et al (2018) use 4 layers of PMD units with 2 skip connections, and Ho et al (2019) identifies the motion vectors of only the critical pixels of the image and uses that to estimate the future motion vectors of those pixels to generate future frames. Autoencoders are a natural framework to use in video prediction because they aggregate both the global context necessary to understand which parts of the scene are moving and why, and local context which helps generate the pixel-by-pixel texture of the future frame (Ronneberger et al, 2015).…”

“…Autoencoders are a natural framework to use in video prediction because they aggregate both the global context necessary to understand which parts of the scene are moving and why, and local context which helps generate the pixel-by-pixel texture of the future frame (Ronneberger et al, 2015). In addition to having an autoencoder, Cho et al (2021) uses a memory network on the scene encoding which memorizes the prototypical motion patterns of different types of objects in order to increase robustness to scenes in different environments. Furthermore, both Cho et al (2021) and add skip connections to the autoencoder in order to add local context (as shown in Ronneberger et al, 2015), and Oliu et al (2018) uses a bijective gated recurrent unit (bGRU) within the autoencoder to allow shared states between the encoder and decoder.…”

“…In addition to having an autoencoder, Cho et al (2021) uses a memory network on the scene encoding which memorizes the prototypical motion patterns of different types of objects in order to increase robustness to scenes in different environments. Furthermore, both Cho et al (2021) and add skip connections to the autoencoder in order to add local context (as shown in Ronneberger et al, 2015), and Oliu et al (2018) uses a bijective gated recurrent unit (bGRU) within the autoencoder to allow shared states between the encoder and decoder. In Liu et al (2018), the dynamical atoms from the input encoding are used by the decoder to predict the next frame by extending their temporal horizon into the future.…”