EgoCap

Abstract

Marker-based and marker-less optical skeletal motion-capture methods use an outside-in arrangement of cameras placed around a scene, with viewpoints converging on the center. They often create discomfort with marker suits, and their recording volume is severely restricted and often constrained to indoor scenes with controlled backgrounds. Alternative suit-based systems use several inertial measurement units or an exoskeleton to capture motion with an inside-in setup, i.e. without external sensors. This makes capture independent of a confined volume, but requires substantial, often constraining, and hard to set up body instrumentation. Therefore, we propose a new method for real-time, marker-less, and egocentric motion capture: estimating the full-body skeleton pose from a lightweight stereo pair of fisheye cameras attached to a helmet or virtual reality headset -an optical inside-in method, so to speak. This allows full-body motion capture in general indoor and outdoor scenes, including crowded scenes with many people nearby, which enables reconstruction in larger-scale activities. Our approach combines the strength of a new generative pose estimation framework for fisheye views with a ConvNet-based body-part detector trained on a large new dataset. It is particularly useful in virtual reality to freely roam and interact, while seeing the fully motion-captured virtual body.

Mo2Cap2 : Real-time Mobile 3D Motion Capture with a Cap-mounted Fisheye Camera

“…Yonemoto et al propose indirect inference of arm and torso poses from arm-only RGB-D footage [70] and Jiang attempted to reconstruct full-body pose by analyzing the egomotion and observed scene [22], but indirect predictions have low confidence and accuracy. A first approach towards direct full-body motion capture from the egocentric perspective was proposed by Rhodin et al [39]. A 3D kinematic skeleton model is optimized to explain 2D features in each of the views of a stereo fisheye camera mounted on head-extensions similar in structure to a selfie stick.…”

“…Our work is the first approach that performs 3D real-time human body pose estimation from a single head-mounted camera. Previous work [39] has demonstrated successful motion capture with a helmetmounted fisheye stereo pair. While their results are promising, their setup has a number of practical disadvantages.…”

“…Capturing a large amount of annotated 3D pose data is already a mammoth task for outside-in setups and it is even harder for egocentric data. Since manual labeling in 3D space is impractical, [39] proposes to use marker-less multi-view motion capture with externally mounted cameras to get 3D annotations.…”

“…However, these setups are expensive, require tedious pre-calibration, and often require pose optimization over the entire sequence, which prevents real-time performance. Most closely related to our approach is the EgoCap [39] system that is based on two head mounted fisheye cameras. While it alleviates the problem of a limited capture volume, the setup is quite heavy and requires uncomfortable, obtrusive large extension-sticks.…”

Efficient Implementation of Gaussian and Laplacian Kernels for Feature Extraction from IP Fisheye Cameras

“…Traffic management at road intersection has been proposed in [10] using network fisheye cameras. Pair of fisheye cameras has been used for pose estimation and motion capture in [11], employing a number of 3D Gaussians to generate a probabilistic body model. Gaussian kernels defined in 3D have also been used for scene modeling, in order to make discrete phenomena, like occlusion, continuous and thus, more easily optimizable [12].…”