Ego-Pose Estimation and Forecasting As Real-Time PD Control

Abstract

Figure 1. Proposed method estimates camera wearer's 3D poses (solid) and forecasts future poses (translucent) in real-time. AbstractWe propose the use of a proportional-derivative (PD) control based policy learned via reinforcement learning (RL) to estimate and forecast 3D human pose from egocentric videos. The method learns directly from unsegmented egocentric videos and motion capture data consisting of various complex human motions (e.g., crouching, hopping, bending, and motion transitions). We propose a video-conditioned recurrent control technique to forecast physically-valid and stable future motions of arbitrary length. We also introduce a value function based fail-safe mechanism which enables our method to run as a single pass algorithm over the video data. Experiments with both controlled and in-the-wild data show that our approach outperforms previous art in both quantitative metrics and visual quality of the motions, and is also robust enough to transfer directly to real-world scenarios. Additionally, our time analysis shows that the combined use of our pose estimation and forecasting can run at 30 FPS, making it suitable for real-time applications. 1

Silhouette-Based Synthetic Data Generation For 3D Human Pose Estimation With A Single Wrist-Mounted 360° Camera

“…In recent years, methods for whole-body 3D pose estimation with more practical settings have been proposed, such as using a wearable camera with a wide viewing angle to capture more body parts [1][2][3][4]11]. Moreover, several studies have achieved 3D pose estimation under the severe condition in which the human body is completely hidden from the camera's line of sight [5,6,12]. Thus far, these existing methods mount the camera on the user's head or chest.…”

“…The 3D human pose estimation network (Fig. 2) is inspired by the method proposed by Yuan and Kitani [5]. The network F takes the input of the equirectangular video frames V 1:T in which the person is silhouetted and predicts the humanoid state z 1:T at each frame.…”

“…Especially, 3D human pose estimation with wearable cameras is key to many important applications, such as lifelogging in terms of medical assistance, monitoring for life support, virtual reality, and sports activity analysis. Several methods that use wearable cameras mounted on the head or chest have been proposed recently [1][2][3][4][5][6]. However, thus far, there is no method with a more practical camera setting, i.e., a single wrist-mounted camera which could be introduced in smartwatches in the future.…”

“…As shown in Fig. 1, we make use of a single 360 • camera (GoPro Max) and a convolutional neural network-based framework following Yuan and Kitani's work [5] to estimate a 3D human pose with only limited visual information. The difficulty is how to prepare the training data; there is no existing dataset for 3D human pose estimation with wrist-mounted cameras.…”

Linear Time Stable PD Controllers for Physics‐based Character Animation

“…We shall refer to this approach as dense factorization (DF) hereafter. Due to its simplicity, the DF method is widely adopted by both the research community and the industry [Cou15,PALvdP18,PBYVDP17,PBVdP16,YK19].…”