VR Facial Animation for Immersive Telepresence Avatars

Abstract

VR Facial Animation is necessary in applications requiring clear view of the face, even though a VR headset is worn. In our case, we aim to animate the face of an operator who is controlling our robotic avatar system. We propose a real-time capable pipeline with very fast adaptation for specific operators. In a quick enrollment step, we capture a sequence of source images from the operator without the VR headset which contain all the important operator-specific appearance information. During inference, we then use the operator keypoint information extracted from a mouth camera and two eye cameras to estimate the target expression and head pose, to which we map the appearance of a source still image. In order to enhance the mouth expression accuracy, we dynamically select an auxiliary expression frame from the captured sequence. This selection is done by learning to transform the current mouth keypoints into the source camera space, where the alignment can be determined accurately. We, furthermore, demonstrate an eye tracking pipeline that can be trained in less than a minute, a time efficient way to train the whole pipeline given a dataset that includes only complete faces, show exemplary results generated by our method, and discuss performance at the ANA Avatar XPRIZE semifinals.

Bimanual telemanipulation with force and haptic feedback through an anthropomorphic avatar system

“…Both hand types thus complement each other. The avatar's head is equipped with two RGB cameras, a microphone, and a small screen displaying the animated face of the operator [27]. It is attached to the upper body using a 6 DoF UFactory xArm for free head movement.…”

iCub3 avatar system: Enabling remote fully immersive embodiment of humanoid robots

“…Touch feedback can have a noticeable effect on the manipulation capabilities of an avatar system (18), but it can also enable teleoperated physical interaction (19), giving rise to social implications. In the context of social presence, other avatar characteristics gain relevance, such as the control of facial expressions (20,21).…”