Virtual Object Replacement Based on Real Environments: Potential Application in Augmented Reality Systems

Abstract

Augmented reality (AR) is an emerging technology that allows users to interact with simulated environments, including those emulating scenes in the real world. Most current AR technologies involve the placement of virtual objects within these scenes. However, difficulties in modeling real-world objects greatly limit the scope of the simulation, and thus the depth of the user experience. In this study, we developed a process by which to realize virtual environments that are based entirely on scenes in the real world. In modeling the real world, the proposed scheme divides scenes into discrete objects, which are then replaced with virtual objects. This enables users to interact in and with virtual environments without limitations. An RGB-D camera is used in conjunction with simultaneous localization and mapping (SLAM) to obtain the movement trajectory of the user and derive information related to the real environment. In modeling the environment, graph-based segmentation is used to segment point clouds and perform object segmentation to enable the subsequent replacement of objects with equivalent virtual entities. Superquadrics are used to derive shape parameters and location information from the segmentation results in order to ensure that the scale of the virtual objects matches the original objects in the real world. Only after the objects have been replaced with their virtual counterparts in the real environment converted into a virtual scene. Experiments involving the emulation of real-world locations demonstrated the feasibility of the proposed rendering scheme. A rock-climbing application scenario is finally presented to illustrate the potential use of the proposed system in AR applications.

Site Experience Enhancement and Perspective in Cultural Heritage Fruition—A Survey on New Technologies and Methodologies Based on a “Four-Pillars” Approach

“…Based on these considerations, the real environment can be added with virtual objects. The same approach is proposed by [53] adopting an RGB-D camera in conjunction with simultaneous localization and mapping (SLAM) to retrieve the user's movement trajectory and obtain information related to the real environment. The SLAM with monocular camera in AR context [54,55], where only some virtual objects are placed within a scene of the real world, removes the limitation of VR and avoids simulating every entity within the scene.…”

“…The SLAM with monocular camera in AR context [54,55], where only some virtual objects are placed within a scene of the real world, removes the limitation of VR and avoids simulating every entity within the scene. In [53], segmentation and shape fitting of objects are used to construct virtual objects to replace (within the virtual world) actual objects encountered (within the real world). Even if the authors claim that this system is suitable only for single-room venues and applicable mainly to static environments, it seems interesting for increasing the experience of visiting museums because the presence of objects can be enriched with others related to the same theme or with multimedia content.…”