Unmasking Communication Partners: A Low-Cost AI Solution for Digitally Removing Head-Mounted Displays in VR-Based Telepresence

Abstract

Face-to-face conversation in Virtual Reality (VR) is a challenge when participants wear head-mounted displays (HMD). A significant portion of a participant's face is hidden and facial expressions are difficult to perceive. Past research has shown that high-fidelity face reconstruction with personal avatars in VR is possible under laboratory conditions with high-cost hardware. In this paper, we propose one of the first low-cost systems for this task which uses only open source, free software and affordable hardware. Our approach is to track the user's face underneath the HMD utilizing a Convolutional Neural Network (CNN) and generate corresponding expressions with Generative Adversarial Networks (GAN) for producing RGBD images of the person's face. We use commodity hardware with low-cost extensions such as 3Dprinted mounts and miniature cameras. Our approach learns end-to-end without manual intervention, runs in real time, and can be trained and executed on an ordinary gaming computer. We report evaluation results showing that our low-cost system does not achieve the same fidelity of research prototypes using high-end hardware and closed source software, but it is capable of creating individual facial avatars with personspecific characteristics in movements and expressions.

Immersive ERP Experiment Based on 3D-Reconstruction Model of Panoramic Images

“…By engaging students in the real situation for operation, it can reduce the experience cost and study cost of the subjects and guide students to carry out deep learning. [5] Enterprise Resource Planning (ERP) was proposed by Gartner Group Company in 1990 and refers to the management platform based on information technology and providing enterprise decision-making levels and employees with systematic management ideas. However, the application of VR technology in ERP experiments is not common, and the ERP teaching in higher education generally uses the traditional teaching methods.…”