DYAN: A Dynamical Atoms-Based Network for Video Prediction

Abstract

The ability to anticipate the future is essential when making real time critical decisions, provides valuable information to understand dynamic natural scenes, and can help unsupervised video representation learning. State-of-art video prediction is based on complex architectures that need to learn large numbers of parameters, are potentially hard to train, slow to run, and may produce blurry predictions. In this paper, we introduce DYAN, a novel network with very few parameters and easy to train, which produces accurate, high quality frame predictions, faster than previous approaches. DYAN owes its good qualities to its encoder and decoder, which are designed following concepts from systems identification theory and exploit the dynamics-based invariants of the data. Extensive experiments using several standard video datasets show that DYAN is superior generating frames and that it generalizes well across domains.

Understanding the perceived quality of video predictions

“…The deterministic models are trained to predict the future frames, exactly as in the ground truth video. The deterministic models we use are PredNet [15], MCnet [14], Future GAN [45] and DYAN [46]. On the other hand, the stochastic models model uncertainty by being trained to predict a distribution of possible futures using noise as an additional input.…”

Comparing Correspondences: Video Prediction with Correspondence-wise Losses

“…Later, Wu et al [92] extended this approach, conditioning predictions on the trajectories of objects that their model segments and tracks. Recent work has used other motion representations, such as factorizing a scene into stationary and moving components [12,85,83], per-pixel kernels [24,86,64,70,47], or Eulerian motion [55]. Work in 3D view synthesis has adopted a similar copy-and-paste approach, known as appearance flow [102,66,66].…”

Exploring Spatial-Temporal Multi-Frequency Analysis for High-Fidelity and Temporal-Consistency Video Prediction

“…The main consideration is to minimize the reconstruction error between the true future frame and the generated future frame. Such models can be classified as direct prediction models [35,46,43,21,3,39,30,38,18,25] and transformationbased prediction models [49,40,37,32]. Direct prediction models predict pixel values of future frames directly.…”