Efficient 3D Reconstruction and Streaming for Group-Scale Multi-Client Live Telepresence

Abstract

SLAMCast 4 ECs Ours 24 ECsFigure 1: Illustration of our novel highly scalable multi-client live telepresence system. While previous approaches are limited to a low number of up to 4 remote exploration clients, our system is capable of providing an immersive telepresence experience within a live-captured high-quality scene reconstruction to more than 24 clients simultaneously without introducing further latency. ABSTRACTSharing live telepresence experiences for teleconferencing or remote collaboration receives increasing interest with the recent progress in capturing and AR/VR technology. Whereas impressive telepresence systems have been proposed on top of on-the-fly scene capture, data transmission and visualization, these systems are restricted to the immersion of single or up to a low number of users into the respective scenarios. In this paper, we direct our attention on immersing significantly larger groups of people into live-captured scenes as required in education, entertainment or collaboration scenarios. For this purpose, rather than abandoning previous approaches, we present a range of optimizations of the involved reconstruction and streaming components that allow the immersion of a group of more than 24 users within the same scene -which is about a factor of 6 higher than in previous work -without introducing further latency or changing the involved consumer hardware setup. We demonstrate that our optimized system is capable of generating high-quality scene reconstructions as well as providing an immersive viewing experience to a large group of people within these live-captured scenes.

Revisiting Milgram and Kishino's Reality-Virtuality Continuum

“…One thread of research in this space is MR telepresence, in which the goal is to capture remote spaces and users and reproduce them elsewhere in real time. Recent papers in this area include Kunert et al (2018), Stotko et al (2019b), and Stotko et al (2019a). Another research thread is what we call world-aware environment generation, in which the goal is to create a virtual environment that possesses some of the same characteristics of the real environment-for example, areas that are not navigable in the real environment are also not navigable in the virtual environment.…”

Efficient 3D Mapping and Modelling of Indoor Scenes with the Microsoft HoloLens: A Survey

“…This can be realised with high frame rates, so that RGB-D cameras also allow an acquisition of dynamic scenes. Addressing both efficient and robust geometry acquisition, KinectFusion (Izadi et al 2011) and its improved variants (Nießner et al 2013;Kähler et al 2016;Dai et al 2017;Stotko et al 2019b) are widely used. However, major limitations of RGB-D cameras are typically given regarding the accuracy of geometry acquisition, which might not meet the standard of indoor surveying applications.…”

3d Indoor Mapping With the Microsoft Hololens: Qualitative and Quantitative Evaluation by Means of Geometric Features

“…Such RGB-D cameras allow for scene capture with high frame rates and are therefore often suitable for acquiring both static and dynamic scenes. Among a diversity of approaches, KinectFusion (Izadi et al, 2011) and respective improvements (Nießner et al, 2013;Kähler et al, 2016;Dai et al, 2017b;Stotko et al, 2019) have become popular methods for fast scene reconstruction. For a detailed survey on 3D scene acquisition with RGB-D cameras, we refer to (Zollhöfer et al, 2018).…”