Generative RGB-D Face Completion for Head-Mounted Display Removal

Abstract

My supervisors, Frank ter Haar and Pablo Cesar, for providing invaluable guidance through their constructive feedback, great patience and extensive knowledge.• My colleagues at the Intelligent Imaging department of TNO, for their exciting suggestions, stimulating feedback and infectious enthusiasm.• My mom and sister, for their endless love, support and encouragement.• My grandparents, for always believing in me.

Multiparty Holomeetings: Toward a New Era of Low-Cost Volumetric Holographic Meetings in Virtual Reality

“…This may not be a serious drawback in entertainment and collaboration scenarios, where the focus is mainly on the content being watched or on the task to be performed, respectively; but it can be a relevant factor in potentially critical or sensitive person-to-person conversations. The authors are fully aware of this issue, but the scientific community is actively investigating effective solutions for replacing in real-time the HMD with estimated facial expressions (e.g., [50]). In addition, expectations are that the nextgeneration eXtended Reality (XR) headsets will be more lightweight and, further, they will block facial expressions and eye contact to a much lesser extent.…”

HEADSET: Human Emotion Awareness under Partial Occlusions Multimodal DataSET

“…While the usage of a synthetic dataset reduces the potential of generalization to real-world data, this approach has been widely used in the literature. For example, for HMD removal/ facial reconstruction, the authors of [34] built a data synthesis pipeline to create a synthetic dataset of RGB-D images with a random pose, ambient illumination, and expression of faces based on the Basel Face Model (BFM) 2017 [15]. To address the aforementioned challenges, our dataset contains additional sequences under HMD occlusions in order to be used as the testing set for the future validation and assessment of such research work.…”

“…The knowledge about the emotion of the participant, who is wearing a headset, might increase the quality of the facial reconstruction. Many studies have focused on HMD removal [7,29,34,54,59], which is referred to as the task of virtual removal of HMD, which fill in the occluded color and geometric information of a user's face. The emergence of new MR glasses with emotion recognition capabilities and transparent displays, such as Meta Quest Pro and Apple Vision Pro, may increase the quality of the facial reconstruction results when removing the headset in such a study.…”

Deep Learning Augmented Realistic Avatars for Social VR Human Representation

“…In this context, GANs have been explored for use in VR applications for HMD removal processes. [34] for example, synthesizes a face from an RGB+depth capture where the face in the original image is occluded with an HMD.…”