A Robust Real-Time 3D Reconstruction Method for Mixed Reality Telepresence

Abstract

Mixed Reality (MR) is a technology which enable to bring a virtual element into the real-world environment. MR intends to improve reality on the virtual world immerse onto real-world space. Occasionally the MR has been improved as the display technologies advanced progressively. In MR collaborative interface context, the local and remote user work together on collaborative task while sense the immersive environment in the cooperative application. User telepresence is an immersive telepresence, where the reconstruction of a human appears in a real-life. Up till now, producing full telepresence of the life-size human body may require a high transmission bandwidth of the internet. Therefore, this paper explores on a robust real-time 3D reconstruction method for MR telepresence. This paper discusses the previous works on the reconstruction method of a full-body human and the existing research works that have proposed the reconstruction methods for telepresence. Besides the 3D reconstruction method, this paper also enlightens our recent finding on the MR framework to transport a full-body human from a local location to a remote location. The MR telepresence will be discussed, as well as the robust 3D reconstruction method which has been implemented with user telepresence feature where the user experiences an accurate 3D representation of a remote person. The paper ends with the discussion and results, MR telepresence with robust 3D reconstruction method to execute user telepresence.

A systematic literature review: Real-time 3D reconstruction method for telepresence system

“…User studies [43,58,70,93] and pilot studies [15] have been conducted and evaluated. [91] evaluate the practicality of the framework for telepresence in live-captured scenes while [58,59] evaluate the user experience of the system.…”

“…When constructing a 3D communication system, choosing appropriate 3D imaging and display technologies is essential. The 3D display methods can be categorized as binocular vision display [59], volume display [36], light field (LF) display [43,60], and holographic display [61,62]. Holographic display is a promising method for giving human eyes all the depth information [26 -30].…”

“…[69] mentioned 3D display technology that has been implemented in the telepresence system can be divided into two main devices, which are projectors [62,[70][71][72][73] and head-mounted devices (HMD) [45,52,58,59,[74][75][76][77]. The projector device's 3D display technologies are an on-stage hologram, autostereoscopic, and holographic projection, while HMDs can be classified into MR headsets and VR headsets.…”

3D Display for 3D Telepresence: A Review

“…A specific application will fall into a different category of the Milgram and Kishino's MR continuum depending on the level of real and virtual data. [29] explores on a robust real-time 3D reconstruction method for MR telepresence. As illustrated in Fig.…”

“…10. MR telepresence user can see the remote user while wearing MR headset [29] An MR framework that is the outcome of the combination of the telepresence with an application for improved collaborative space exploration were presented by [31] and Fig. 11 shows two users who are geologically distinct can meet each other.…”