Dynamic Rendering of Remote Indoor Environments Using Real-Time Point Cloud Data

Abstract

Modern color and depth (RGB-D) sensing systems are capable of reconstructing convincing virtual representations of real world environments. These virtual reconstructions can be used as the foundation for virtual reality (VR) and augmented reality environments due to their high-quality visualizations. However, a main limitation of modern virtual reconstruction methods is the time it takes to incorporate new data and update the virtual reconstruction. This delay prevents the reconstruction from accurately rendering dynamic objects or portions of the environment (like an engineer performing an inspection of a machinery or laboratory space). The authors propose a multisensor method to dynamically capture objects in an indoor environment. The method automatically aligns the sensors using modern image homography techniques, leverages graphics processing units (GPUs) to process the large number of independent RGB-D data points, and renders them in real time. Incorporating and aligning multiple sensors allows a larger area to be captured from multiple angles, providing a more complete virtual representation of the physical space. Performing processing on GPU's leverages the large number of processing cores available to minimize the delay between data capture and rendering. A case study using commodity RGB-D sensors, computing hardware, and standard transmission control protocol internet connections is presented to demonstrate the viability of the proposed method.

Toward immersive communications in 6G

“…Taking holographic teleconference as an example, as multiple participants can join the teleconference from different locations, multi-source synchronization is necessary for them to have good quality of experience (QoE) in holographic communication. Otherwise, a part of the rendered hologram can be slightly ahead or behind relative to the rest of hologram for some users, resulting in poor QoE (Lesniak and Tucker, 2018). Moreover, holographic communication can involve multi-sensory information, e.g., the haptic, audio, and video information (Taleb et al, 2021).…”

Toward Immersive Communications in 6G

“…Taking holographic teleconference as an example, as multiple participants can join the teleconference from different locations, multisource synchronization is necessary for them to have good quality of experience (QoE) in holographic communication. Otherwise, a part of the rendered hologram can be slightly ahead or behind relative to the rest of hologram for some users, resulting in poor QoE [175]. Moreover, holographic communication can involve multi-sensory information, e.g., the haptic, audio, and video information [33].…”

Impact of resolution, colour, and motion on object identification in digital twins from robot sensor data

“…They proposed a novel approach for rendering a PCVE from robot sensor data. A number of authors have extended this idea using different approaches to render PVCEs ( Schwarz et al, 2017 ; Lesniak and Tucker, 2018 ; Valenzuela-Urrutia et al, 2019 ), Voxel Virtual Environments (VVEs) ( Mossel and Kroeter, 2017 ; Zhou and Tuzel, 2018 ; Stotko et al, 2019 ; Li et al, 2022 ), and VEs composed of precomposed objects based on automated recognition of objects from the point cloud data ( Zhou et al, 2020 ).…”

“…Faithful rendering of point cloud data results in the highest fidelity digital twins. Indeed, it is a typical and reasonable approach for remote scene reconstruction for dynamic scenes where real-time point cloud to model rendering is almost impossible ( Lesniak and Tucker, 2018 ). However, doing so is not without issue.…”