Learning to Predict Diverse Human Motions from a Single Image via Mixture Density Networks

Abstract

Human motion prediction, which plays a key role in computer vision, generally requires a past motion sequence as input. However, in real applications, a complete and correct past motion sequence can be too expensive to achieve. In this paper, we propose a novel approach to predict future human motions from a much weaker condition, i.e., a single image, with mixture density networks (MDN) modeling. Contrary to most existing deep human motion prediction approaches, the multimodal nature of MDN enables the generation of diverse future motion hypotheses, which well compensates for the strong stochastic ambiguity aggregated by the single input and human motion uncertainty. In designing the loss function, we further introduce an energybased prior over learnable parameters of MDN to maintain motion coherence, as well as improve the prediction accuracy. Our trained model directly takes an image as input and generates multiple plausible motions that satisfy the given condition. Extensive experiments on two standard benchmark datasets demonstrate the effectiveness of our method, in terms of prediction diversity and accuracy.

Prediction of Social Dynamic Agents and Long-Tailed Learning Challenges: A Survey

“…Motion prediction is a regression task, where a series of coordinates that correspond to an agent's future pose or location are predicted using their past pose or location, sometimes in combination with other features like ego video (e.g., Adeli et al, 2021), maps (e.g., Salzmann et al, 2021), head orientation (e.g., Haddad & Lam, 2021), body positioning (e.g., , GPS location, (e.g., Sadeghian et al, 2018b), and/or extracted visual features from cropped images (e.g., Haddad & Lam, 2021) of the agents in the scene. Multimodality in motion prediction is a large area of interest in both trajectory (e.g., Dong et al, 2021;Kosaraju et al, 2019;Gu et al, 2022) and pose (e.g., Fragkiadaki et al, 2017;Gu et al, 2021;Yan et al, 2018) 2020) and Korbmacher and Tordeux (2021).…”