Motion Prediction Under Multimodality with Conditional Stochastic Networks

Abstract

Given a visual history, multiple future outcomes for a video scene are equally probable, in other words, the distribution of future outcomes has multiple modes. Multimodality is notoriously hard to handle by standard regressors or classifiers: the former regress to the mean and the latter discretize a continuous high dimensional output space. In this work, we present stochastic neural network architectures that handle such multimodality through stochasticity: future trajectories of objects, body joints or frames are represented as deep, non-linear transformations of random (as opposed to deterministic) variables. Such random variables are sampled from simple Gaussian distributions whose means and variances are parametrized by the output of convolutional encoders over the visual history. We introduce novel convolutional architectures for predicting future body joint trajectories that outperform fully connected alternatives [29]. We introduce stochastic spatial transformers through optical flow warping for predicting future frames, which outperform their deterministic equivalents [17]. Training stochastic networks involves an intractable marginalization over stochastic variables. We compare various training schemes that handle such marginalization through a) straightforward sampling from the prior, b) conditional variational autoencoders [23,29], and, c) a proposed K-best-sample loss that penalizes the best prediction under a fixed " prediction budget". We show experimental results on object trajectory prediction, human body joint trajectory prediction and video prediction under varying future uncertainty, validating quantitatively and qualitatively our architectural choices and training schemes.

Compositional Video Prediction

“…The uncertainty is usually encoded as a sequence of latent variables, which are then used in a generative model such as GAN [12] based [27,34,5,31], or, similar to ours, VAE [20] based [35,6]. These methods [11,6,41] often leverage an input sequence instead of a single frame, which helps reduce the ambiguities. Further, the latent variables are either per-timestep [6], or global [1,41] whereas our model leverages a global latent variable, which in turn induces per-timestep variables.…”

A Review on Deep Learning Techniques for Video Prediction

“…Authors focused on evaluating the linearization properties, yet the model was not contrasted to previous works. Extending [92], [186], Fragkiadaki et al [177] proposed several architectural changes and training schemes to handle marginalization over stochastic variables, such as sampling from the prior and variational inference. Their stochastic ED architecture predicts future optical flow, i.e., dense pixel motion field, used to spatially transform the current frame into the next frame prediction.…”

LARNet: Latent Action Representation for Human Action Synthesis

“…Using future frames as ground-truth leads to conditioned supervised learning approach which gives better results in contrast to unconditional video generation [8,18,28,39]. GAN based approaches often relies on a sequence of input frames as priors to reduce ambiguity [15,19,62,71]. Our approach uses only the first input frame and action class name as prior for the prediction task similar to [28,60].…”