Free-viewpoint AR human-motion reenactment based on a single RGB-D video stream

Abstract

When observing a person (an actor) performing or demonstrating some activity for the purpose of learning the action, it is best for the viewers to be present at the same time and place as the actor. Otherwise, a video must be recorded. However, conventional video only provides two-dimensional (2D) motion, which lacks the original third dimension of motion. In the presence of some ambiguity, it may be hard for the viewer to comprehend the action with only two dimensions, making it harder to learn the action. This paper proposes an augmented reality system to reenact such actions at any time the viewer wants, in order to aid comprehension of 3D motion. In the proposed system, a user first captures the actor's motion and appearance, using a single RGB-D camera. Upon a viewer's request, our system displays the motion from an arbitrary viewpoint using a rough 3D model of the subject, made up of cylinders, and selecting the most appropriate textures based on the viewpoint and the subject's pose. We evaluate the usefulness of the system and the quality of the displayed images by user study.

Increasing pose comprehension through augmented reality reenactment

“…We show that by viewing the AR reenactments, users are more easily able to comprehend ambiguous poses. We also subjectively survey the visual quality of the synthesized reenactment, as well as its applicability, and compare it to our previous work [9]. We found that while the visual quality is not at the level of standard video, it is much improved, and is enough to be easily comprehensible.…”

“…Our previous system [9] uses a model-based approach as well, using a cylinder as each body part instead of its detailed shape model. In this article, we newly [18] to obtain a body model in order to increase the quality of the output.…”

“…We gave users a survey on the quality and applicability of the system, comparing it to the quality and applicability of our previous system [9]. First, users were shown the "Exercise" motion sequence rendered using our previous method, i.e., using cylinders for each body part, as in Fig.…”

“…In this work, we propose the concept of a reenactment [9]. A reenactment is a rendering of a human actor, e.g., an instructor or a performer, and is presented to viewers through their smartphones, tablet PCs, and head-mounted displays.…”

“…-We introduce the concept of reenactments, originally proposed in our previous paper [9]. Reenactments potentially have a wide range of applications, including training motions, watching performances, recording sports, etc.…”