Learning to Generate Long-term Future via Hierarchical Prediction

Villegas, Ruben; Yang, Jimei; Zou, Yuliang; Sohn, Sungryull; Lin, Xunyu; Lee, Honglak

doi:10.48550/arxiv.1704.05831

Cited by 20 publications

(37 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We leverage the 3D Dynamic Prediction (3DDP) network to predict 3DMM dynamics, which can be used for 3D face reconstruction and texture mapping to provide valuable information for face generation. Different with the method [52] that only learns to predict a reasonable future from the initial state, we focus on three tasks, i.e., face video retargeting, video prediction the same as [52] and target-driven video prediction that "imagines" the temporal changes via completing missing frames between the given source and target face images. For the retargeting task, we use a reference video to offer the sequential changes, rather than using 3DDP to predict the future changes.…”

Section: B 3d Dynamic Predictionmentioning

confidence: 99%

Image-to-Video Generation via 3D Facial Dynamics

Zou

Zhao³

et al. 2022

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

We present a versatile model, FaceAnime, for various video generation tasks from still images. Video generation from a single face image is an interesting problem and usually tackled by utilizing Generative Adversarial Networks (GANs) to integrate information from the input face image and a sequence of sparse facial landmarks. However, the generated face images usually suffer from quality loss, image distortion, identity change, and expression mismatching due to the weak representation capacity of the facial landmarks. In this paper, we propose to "imagine" a face video from a single face image according to the reconstructed 3D face dynamics, aiming to generate a realistic and identity-preserving face video, with precisely predicted pose and facial expression. The 3D dynamics reveal changes of the facial expression and motion, and can serve as a strong prior knowledge for guiding highly realistic face video generation. In particular, we explore face video prediction and exploit a well-designed 3D dynamic prediction network to predict a 3D dynamic sequence for a single face image. The 3D dynamics are then further rendered by the sparse texture mapping algorithm to recover structural details and sparse textures for generating face frames. Our model is versatile for various AR/VR and entertainment applications, such as face video retargeting and face video prediction. Superior experimental results have well demonstrated its effectiveness in generating high-fidelity, identitypreserving, and visually pleasant face video clips from a single source face image.

show abstract

Section: B 3d Dynamic Predictionmentioning

confidence: 99%

Image-to-Video Generation via 3D Facial Dynamics

Zou

Zhao³

et al. 2022

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

show abstract

“…Subsequent work that explicitly considered such basic image structure yielded improved results [96,104]; however, prediction horizons were little extended. To support longer term predictions, work has relied on higher-level image analysis (e.g., body keypoint and motion analysis [114,158,159]). Other work has made use of stochastic generative models to capture uncertainty [5,84,157].…”

Section: Overview Of Video Predictive Understandingmentioning

confidence: 99%

Review of Video Predictive Understanding: Early Action Recognition and Future Action Prediction

Zhao¹,

Wildes²

2021

Preprint

View full text Add to dashboard Cite

Video predictive understanding encompasses a wide range of efforts that are concerned with the anticipation of the unobserved future from the current as well as historical video observations. Action prediction is a major sub-area of video predictive understanding and is the focus of this review. This sub-area has two major subdivisions: early action recognition and future action prediction. Early action recognition is concerned with recognizing an ongoing action as soon as possible. Future action prediction is concerned with the anticipation of actions that follow those previously observed. In either case, the causal relationship between the past, current and potential future information is the main focus.Various mathematical tools such as Markov Chains, Gaussian Processes, Auto-Regressive modeling and Bayesian recursive filtering are widely adopted jointly with computer vision techniques for these two tasks.However, these approaches face challenges such as the curse of dimensionality, poor generalization and constraints from domain specific knowledge. Recently, structures that rely on deep convolutional neural networks and recurrent neural networks have been extensively proposed for improving performance of existing vision tasks, in general, and action prediction tasks, in particular. However, they have their own shortcomings, e.g., reliance on massive training data and lack of strong theoretical underpinnings. In this survey, we start by introducing the major sub-areas of the broad area of video predictive understanding, which recently have received intensive attention and proven to have practical value. Next, a thorough review of various early action recognition and future action prediction algorithms are provided with suitably organized divisions. Finally, we conclude our discussion with future research directions.

show abstract

“…The above-mentioned approaches all modeled each human as one rigid object, ignoring the local movements of human body joints. To pay attention on both moving speed and pose variances of people, the approaches [21], [22] took human skeletons as the input to RNN and modeled human movements in this way. Recently, paper [23] proposed to divide human skeletons into five parts and feed them into five independent RNNs for feature extraction.…”

Section: B Human Action Recognition In Videosmentioning

confidence: 99%

A Hierarchical Spatio-Temporal Graph Convolutional Neural Network for Anomaly Detection in Videos

Zeng,

Jiang,

Ding

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep learning models have been widely used for anomaly detection in surveillance videos. Typical models are equipped with the capability to reconstruct normal videos and evaluate the reconstruction errors on anomalous videos to indicate the extent of abnormalities. However, existing approaches suffer from two disadvantages. Firstly, they can only encode the movements of each identity independently, without considering the interactions among identities which may also indicate anomalies. Secondly, they leverage inflexible models whose structures are fixed under different scenes, this configuration disables the understanding of scenes. In this paper, we propose a Hierarchical Spatio-Temporal Graph Convolutional Neural Network (HST-GCNN) to address these problems, the HSTGCNN is composed of multiple branches that correspond to different levels of graph representations. High-level graph representations encode the trajectories of people and the interactions among multiple identities while low-level graph representations encode the local body postures of each person. Furthermore, we propose to weightedly combine multiple branches that are better at different scenes. An improvement over single-level graph representations is achieved in this way. An understanding of scenes is achieved and serves anomaly detection. High-level graph representations are assigned higher weights to encode moving speed and directions of people in low-resolution videos while low-level graph representations are assigned higher weights to encode human skeletons in high-resolution videos. Experimental results show that the proposed HSTGCNN significantly outperforms current state-ofthe-art models on four benchmark datasets (UCSD Pedestrian, ShanghaiTech, CUHK Avenue and IITB-Corridor) by using much less learnable parameters.

show abstract

Learning to Generate Long-term Future via Hierarchical Prediction

Cited by 20 publications

References 0 publications

Image-to-Video Generation via 3D Facial Dynamics

Image-to-Video Generation via 3D Facial Dynamics

Review of Video Predictive Understanding: Early Action Recognition and Future Action Prediction

A Hierarchical Spatio-Temporal Graph Convolutional Neural Network for Anomaly Detection in Videos

Contact Info

Product

Resources

About