SME-Net: Sparse Motion Estimation for Parametric Video Prediction Through Reinforcement Learning

Ho, Yung-Han; Cho, Chuan-Yuan; Jin, Guo-Lun; Peng, Wen-Hsiao

doi:10.1109/iccv.2019.01056

Cited by 13 publications

(10 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Castrejón et al also observed that high capacity models improve generation results. Other work has introduced compositional models [97] and sparse predictions [87,31,32]. Our approach is complementary to this line of work; in our experiments we obtain benefits from using our loss in VAE-based video prediction [13,84].…”

Section: Related Workmentioning

confidence: 77%

Comparing Correspondences: Video Prediction with Correspondence-wise Losses

Geng¹,

Owens²

2021

Preprint

View full text Add to dashboard Cite

Today's image prediction methods struggle to change the locations of objects in a scene, producing blurry images that average over the many positions they might occupy. In this paper, we propose a simple change to existing image similarity metrics that makes them more robust to positional errors: we match the images using optical flow, then measure the visual similarity of corresponding pixels. This change leads to crisper and more perceptually accurate predictions, and can be used with any image prediction network. We apply our method to predicting future frames of a video, where it obtains strong performance with simple, off-the-shelf architectures.

show abstract

Section: Related Workmentioning

confidence: 77%

Comparing Correspondences: Video Prediction with Correspondence-wise Losses

Geng¹,

Owens²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…We also see in Figures 12,22,26,23,27,28,29,30,24, 25 the performance of different competing methods versus NUQ on the BAIR push dataset, trained with 2000 samples. The figures reveal that our method captures the motion of the robot arm, reasonably well, compared to competing methods.…”

Section: F Qualitative Resultsmentioning

confidence: 99%

“…Another line of work in frame prediction seeks to decouple the video into static and moving components [55,14,40,26,61,21]. Some of these approaches are deterministic, others stochastic.…”

Section: Related Workmentioning

confidence: 99%

A Hierarchical Variational Neural Uncertainty Model for Stochastic Video Prediction

Chatterjee¹,

Ahuja²,

Cherian³

2021

Preprint

View full text Add to dashboard Cite

Predicting the future frames of a video is a challenging task, in part due to the underlying stochastic real-world phenomena. Prior approaches to solve this task typically estimate a latent prior characterizing this stochasticity, however do not account for the predictive uncertainty of the (deep learning) model. Such approaches often derive the training signal from the mean-squared error (MSE) between the generated frame and the ground truth, which can lead to sub-optimal training, especially when the predictive uncertainty is high. Towards this end, we introduce Neural Uncertainty Quantifier (NUQ) -a stochastic quantification of the model's predictive uncertainty, and use it to weigh the MSE loss. We propose a hierarchical, variational framework to derive NUQ in a principled manner using a deep, Bayesian graphical model. Our experiments on four benchmark stochastic video prediction datasets show that our proposed framework trains more effectively compared to the state-of-the-art models (especially when the training sets are small), while demonstrating better video generation quality and diversity against several evaluation metrics.

show abstract

“…Hang et al [10] introduced a confidence-aware warping operator to predict occluded area and disoccluded area separately. Ho et al [15] proposed a parametric video prediction approach based on a sparse motion field.…”

Section: Related Workmentioning

confidence: 99%

Future Video Synthesis with Object Motion Prediction

Wu¹,

Gao²,

Park

et al. 2020

Preprint

View full text Add to dashboard Cite

We present an approach to predict future video frames given a sequence of continuous video frames in the past. Instead of synthesizing images directly, our approach is designed to understand the complex scene dynamics by decoupling the background scene and moving objects. The appearance of the scene components in the future is predicted by non-rigid deformation of the background and affine transformation of moving objects. The anticipated appearances are combined to create a reasonable video in the future. With this procedure, our method exhibits much less tearing or distortion artifact compared to other approaches. Experimental results on the Cityscapes and KITTI datasets show that our model outperforms the state-of-the-art in terms of visual quality and accuracy.

show abstract

SME-Net: Sparse Motion Estimation for Parametric Video Prediction Through Reinforcement Learning

Cited by 13 publications

References 8 publications

Comparing Correspondences: Video Prediction with Correspondence-wise Losses

Comparing Correspondences: Video Prediction with Correspondence-wise Losses

A Hierarchical Variational Neural Uncertainty Model for Stochastic Video Prediction

Future Video Synthesis with Object Motion Prediction

Contact Info

Product

Resources

About