3D Ego-Pose Estimation via Imitation Learning

Yuan, Ye; Kitani, Kris M.

doi:10.1007/978-3-030-01270-0_45

Cited by 65 publications

(56 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Jiang and Grauman [52] reconstruct full body pose from footage taken from a camera worn on the chest by estimating egomotion from the observed scene, but their estimates lack accuracy and have high uncertainty. Yuan et al [53], [54] instead explores a different solution by moving away from kinematics-based representations and using a control-based representation of humanoid motion, commonly used in robotics. A step towards dealing with large parts of the body not being observable was proposed in [55] but for external camera viewpoints.…”

Section: Monocular 3d Pose Estimation From An External Cameramentioning

confidence: 99%

SelfPose: 3D Egocentric Pose Estimation From a Headset Mounted Camera

Tomè

Alldieck

Peluse

et al. 2023

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

We present a new solution to egocentric 3D body pose estimation from monocular images captured from a downward looking fish-eye camera installed on the rim of a head mounted virtual reality device. This unusual viewpoint leads to images with unique visual appearance, characterized by severe self-occlusions and strong perspective distortions that result in a drastic difference in resolution between lower and upper body. We propose a new encoder-decoder architecture with a novel multi-branch decoder designed specifically to account for the varying uncertainty in 2D joint locations. Our quantitative evaluation, both on synthetic and real-world datasets, shows that our strategy leads to substantial improvements in accuracy over state of the art egocentric pose estimation approaches. To tackle the severe lack of labelled training data for egocentric 3D pose estimation we also introduced a large-scale photo-realistic synthetic dataset. xR-EgoPose offers 383K frames of high quality renderings of people with diverse skin tones, body shapes and clothing, in a variety of backgrounds and lighting conditions, performing a range of actions. Our experiments show that the high variability in our new synthetic training corpus leads to good generalization to real world footage and to state of the art results on real world datasets with ground truth. Moreover, an evaluation on the Human3.6M benchmark shows that the performance of our method is on par with top performing approaches on the more classic problem of 3D human pose from a third person viewpoint.

show abstract

Section: Monocular 3d Pose Estimation From An External Cameramentioning

confidence: 99%

SelfPose: 3D Egocentric Pose Estimation From a Headset Mounted Camera

Tomè

Alldieck

Peluse

et al. 2023

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…For ego-pose estimation, we can see our approach outperforms other baselines in terms of both pose-based metric (pose error) and physics-based metrics (velocity error, acceleration, number of resets). We find that VGAIL [67] is often unable to learn a stable control policy from the training data due to frequent falling, which results in the high number of resets and large acceleration. For ego-pose forecasting, our method is more accurate than other methods for both short horizons and long horizons.…”

Section: Resultsmentioning

confidence: 98%

“…It has been shown that if the task of pose estimation can be limited to a single mode of action such as running or walking, it is possible to estimate a physically-valid pose sequence. Recent work by Yuan and Kitani [67] has formulated egocentric pose estimation as a Markov decision process (MDP): a humanoid agent driven by a control policy with visual input to generate a pose sequence inside a physics simulator. They use generative adversarial imitation learning (GAIL [14]) to solve for the optimal control policy.…”

Section: Introductionmentioning

confidence: 99%

“…As a result, this irregularity in control actions causes the humanoid to lose balance and fall in the physics environment, preventing the method from providing any pose estimates. The control-based method proposed in [67] prevented falling by fine-tuning the policy at test time as a batch process. As a result, this prohibits its use in streaming or real-time applications.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Ego-Pose Estimation and Forecasting As Real-Time PD Control

Yuan

Kitani

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

102

View full text Add to dashboard Cite

Figure 1. Proposed method estimates camera wearer's 3D poses (solid) and forecasts future poses (translucent) in real-time. AbstractWe propose the use of a proportional-derivative (PD) control based policy learned via reinforcement learning (RL) to estimate and forecast 3D human pose from egocentric videos. The method learns directly from unsegmented egocentric videos and motion capture data consisting of various complex human motions (e.g., crouching, hopping, bending, and motion transitions). We propose a video-conditioned recurrent control technique to forecast physically-valid and stable future motions of arbitrary length. We also introduce a value function based fail-safe mechanism which enables our method to run as a single pass algorithm over the video data. Experiments with both controlled and in-the-wild data show that our approach outperforms previous art in both quantitative metrics and visual quality of the motions, and is also robust enough to transfer directly to real-world scenarios. Additionally, our time analysis shows that the combined use of our pose estimation and forecasting can run at 30 FPS, making it suitable for real-time applications. 1

show abstract

“…The camera motions are taken from the video and the scene itself, using CNN'sbased feature extractors, taking into account the preceding frames and poses. Recently, a new approach [6,18] is reported to prophesy the 3D-pose through the egocentric video. In this approach, the imitation method of learning is used for understanding a policy controller for the extrapolation of a pose.…”

Section: Introductionmentioning

confidence: 99%

Human Action Recognition through the First-Person Point of view, Case Study Two Basic Task

Almasi¹,

Fathi²,

Ghaeinian³

et al. 2019

IJCA

View full text Add to dashboard Cite

In this study, a human motion dataset is built and developed based on indoors and outdoors actions through a bounded-onhead camera and Xsens for tracking the motions. The key point here to structuring the dataset is utilized to set the sequence of a Deep Neural Network and order an arrangement of frames in the performed task (washing, eating, etc.). As a final point, a 3D modeling of the person suggested at every frame centered with the comparable structure of the first network. More than 120,000 frames constructed the dataset, taken from 7 different people, each one acting out different tasks in diverse indoor and outdoor scenarios. The sequences of every video frame were 3D synchronized and segmented 23 parts.

show abstract

3D Ego-Pose Estimation via Imitation Learning

Cited by 65 publications

References 21 publications

SelfPose: 3D Egocentric Pose Estimation From a Headset Mounted Camera

SelfPose: 3D Egocentric Pose Estimation From a Headset Mounted Camera

Ego-Pose Estimation and Forecasting As Real-Time PD Control

Human Action Recognition through the First-Person Point of view, Case Study Two Basic Task

Contact Info

Product

Resources

About