Physics-based Human Motion Estimation and Synthesis from Videos

Xie, Kevin; Wang, Tingwu; Iqbal, Umar; Guo, Yunrong; Fidler, Sanja; Shkurti, Florian

doi:10.1109/iccv48922.2021.01133

Cited by 55 publications

(22 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Future extensions of this work should therefore explore how to best use past frames and inputs. This could be coupled with a physics based approach, either as part of a controller [73] or using explicit physical losses [68] in L D . Finally, another interesting direction is the use of more effective parameterizations for the per-step weights [13,25].…”

Section: Discussionmentioning

confidence: 99%

Learning to Fit Morphable Models

Choutas¹,

Bogo²,

Shen³

et al. 2021

Preprint

View full text Add to dashboard Cite

Figure 1. Accurate head and hand poses are becoming readily available on AR and VR headsets. One current challenge to increase immersion in virtual worlds consists in fitting 3D body models to these sparse measurements (top left) which is a problem often tackled using regression followed by iterative mathematical optimization (top right). While these approaches have demonstrated their effectiveness on many problems, they require significant amounts of manual labor to work at speed and quality. Using machine learning for continuous optimization offers the opportunity to overcome this hurdle by learning the priors directly from the data and by leveraging optimized backend implementations (bottom left). In this way, we can construct effective models, without the biases introduced by the manually defined energy terms. The resulting models are fast, able to tightly fit the data (bottom right, ground-truth mesh in blue) and can easily be trained for different problem setups. Note that all results are estimated independently per-frame.

show abstract

Section: Discussionmentioning

confidence: 99%

Learning to Fit Morphable Models

Choutas¹,

Bogo²,

Shen³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The common metrics of the Mean Per Joint Position Error (MPJPE) and the MPJPE after rigid alignment of the prediction with ground truth using Procrustes Analysis (MPJPE-PA) are used to evaluate joint accuracy. To evaluate physical plausibility, we use the metrics proposed in [51] and [62] to measure motion jitter and foot contact. e S is the difference in joint velocity magnitude between the ground truth motion and the predicted motion.…”

Section: Metricsmentioning

confidence: 99%

“…To improve motion quality and physical plausibility, a few works focus on capturing human motion using physicsbased constraints. [46,50,51,57,62] incorporate physical laws as soft constraints in numerical optimization framework and reduce artifacts. To make optimization be tractable, they can only adopt simple and differentiable physical models, which may result in high approximation errors.…”

Section: Introductionmentioning

confidence: 99%

Neural MoCon: Neural Motion Control for Physically Plausible Human Motion Capture

Huang¹,

Peng²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Due to the visual ambiguity, purely kinematic formulations on monocular human motion capture are often physically incorrect, biomechanically implausible, and can not reconstruct accurate interactions. In this work, we focus on exploiting the high-precision and non-differentiable physics simulator to incorporate dynamical constraints in motion capture. Our key-idea is to use real physical supervisions to train a target pose distribution prior for sampling-based motion control to capture physically plausible human motion. To obtain accurate reference motion with terrain interactions for the sampling, we first introduce an interaction constraint based on SDF (Signed Distance Field) to enforce appropriate ground contact modeling. We then design a novel two-branch decoder to avoid stochastic error from pseudo ground-truth and train a distribution prior with the non-differentiable physics simulator. Finally, we regress the sampling distribution from the current state of the physical character with the trained prior and sample satisfied target poses to track the estimated reference motion. Qualitative and quantitative results show that we can obtain physically plausible human motion with complex terrain interactions, human shape variations, and diverse behaviors. More information can be found at https:// www.yangangwang.com/papers/HBZ-NM-2022-03.html

show abstract

“…the camera. To address the lack of translation, recent methods start to estimate human meshes in the camera coordinates [33,36,53,58,74,77,84,98,106,108,110]. Several approaches recover the absolute translation of the person using an optimization framework [62-64, 80, 107].…”

Section: Related Workmentioning

confidence: 99%

“…A few methods exploit various scene constraints during the optimization process to improve depth prediction [95,106]. Alternatively, recent approaches use physics-based constraints to ensure the physical plausibility of the estimated poses [12,34,84,98,104]. Iqbal et al [32] exploit a limblength constraint to recover the absolute translation of the person using a 2.5D representation.…”

Section: Related Workmentioning

confidence: 99%