Immersive Distributed Design Through Real-Time Capture, Translation, and Rendering of Three-Dimensional Mesh Data1

Abstract

With design teams becoming more distributed, the sharing and interpreting of complex data about design concepts/prototypes and environments have become increasingly challenging. The size and quality of data that can be captured and shared directly affects the ability of receivers of that data to collaborate and provide meaningful feedback. To mitigate these challenges, the authors of this work propose the real-time translation of physical objects into an immersive virtual reality environment using readily available red, green, blue, and depth (RGB-D) sensing systems and standard networking connections. The emergence of commercial, off-the-shelf RGB-D sensing systems, such as the Microsoft Kinect, has enabled the rapid three-dimensional (3D) reconstruction of physical environments. The authors present a method that employs 3D mesh reconstruction algorithms and real-time rendering techniques to capture physical objects in the real world and represent their 3D reconstruction in an immersive virtual reality environment with which the user can then interact. Providing these features allows distributed design teams to share and interpret complex 3D data in a natural manner. The method reduces the processing requirements of the data capture system while enabling it to be portable. The method also provides an immersive environment in which designers can view and interpret the data remotely. A case study involving a commodity RGB-D sensor and multiple computers connected through standard TCP internet connections is presented to demonstrate the viability of the proposed method.

A shared realities workflow for interactive design using virtual reality and three-dimensional depth sensing

“…Various applications of depth sensors have included occupancy detection and profiling for building energy management, 26 scanning of large environments to generate virtual representations in interactive 3D environments, 27 and capturing physical objects for representation in a real-time user-interactive, immersive VR environment. 28…”

“…Various applications of depth sensors have included occupancy detection and profiling for building energy management, 26 scanning of large environments to generate virtual representations in interactive 3D environments, 27 and capturing physical objects for representation in a realtime user-interactive, immersive VR environment. 28 The traditional methods of 3D user tracking with depth sensors, via Open Natural Interaction (OpenNI) and Kinect Software Development Kit (SDK) skeleton tracking, proved to be too limited for our application due to restrictions on sensor placement and the number of users that could be continuously tracked. Most gestural interaction only requires tracking each user's head and hands which can be fairly easily deduced directly from a user point cloud, which itself is derived from the overall point cloud via a simple grouping algorithm.…”