Capture, Reconstruction, and Representation of the Visual Real World for Virtual Reality

Richardt, Christian; Tompkin, James; Wetzstein, Gordon

doi:10.1007/978-3-030-41816-8_1

Cited by 22 publications

(16 citation statements)

References 113 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus the methods are subdivided into three categories: without, explicit and implicit geometric proxy restoration. Richardt et al [22] updated a similar categorization in the form of continuum which was proposed by [25]. The synthesis approach…”

Section: Scene Synthesis Methodsmentioning

confidence: 99%

A Simulation System for Scene Synthesis in Virtual Reality

Liu

Mantel

Schweiger³

et al. 2021

Virtual Reality and Mixed Reality

View full text Add to dashboard Cite

Real-world scene synthesis can be realized through view synthesis or 3D reconstruction methods. While industrial and commercial demands emerge for real-world scene synthesis in virtual reality (VR) using head-mounted displays (HMDs), the methods in the literature generally do not target specific display devices. To meet the rising demands, we propose a simulation system to evaluate scene synthesis methods in VR. Our system aims at providing the full pipeline of scene capturing, processing, rendering, and evaluation. The capturing module provides various input dataset formulations. The processing and rendering module integrates three representative scene synthesis methods with a voluntary performance-aid option. Finally, the evaluation module supports traditional metrics as well as perception-based metrics. An experiment demonstrates the use of our system for identifying the best capturing strategy among the three degrees of foveation tested. As can be expected, the FovVideoVDP metric (based on a model of the human visual system) finds the highest degree of foveation giving best results. The three other quality metrics from the evaluation module (which use features to measure similarity) confirm that result. The synthetic scenes in the experiment can run in VR with an average latency of 5.9 ms for the two selected scenarios across the tested methods on Nvidia GTX 2080 ti.

show abstract

Section: Scene Synthesis Methodsmentioning

confidence: 99%

A Simulation System for Scene Synthesis in Virtual Reality

Liu

Mantel

Schweiger³

et al. 2021

Virtual Reality and Mixed Reality

View full text Add to dashboard Cite

show abstract

“…Our perception of the world depends on the integration of information from multiple senses. Several works have reviewed the influence of individual senses in isolation, including sight [167], sound [191], or touch [102], which are the three key modalities in VR (see Table 5). Several surveys exist focusing on particular aspects of VR.…”

Section: Related Surveysmentioning

confidence: 99%

Multimodality in VR: A Survey

et al. 2022

View full text Add to dashboard Cite

Virtual reality (VR) is rapidly growing, with the potential to change the way we create and consume content. In VR, users integrate multimodal sensory information they receive, to create a unified perception of the virtual world. In this survey, we review the body of work addressing multimodality in VR, and its role and benefits in user experience, together with different applications that leverage multimodality in many disciplines. These works thus encompass several fields of research, and demonstrate that multimodality plays a fundamental role in VR; enhancing the experience, improving overall performance, and yielding unprecedented abilities in skill and knowledge transfer.

show abstract

“…RGB-D image registration techniques are surveyed in [7] and simultaneous localization and mapping (SLAM) methods are reviewed in [35], both of which are out of scope in this survey. The subject of capturing and scanning real world objects has been extensively surveyed with a focus on RGB-D images in [8] and VR-centered capturing in [9]. The capturing and scanning of point clouds is also out of scope in this survey.…”

Section: Related Surveys and Reviewsmentioning

confidence: 99%

“…with RGB-depth (RGB-D) cameras -reviewed in various publications [7]- [9] -has seen a lot of research after the seminal publication introducing KinectFusion [10]. Many of the fusion-based systems reconstruct the scenes directly into mesh representations with an intermediate scene description based on signed distance fields (SDF) at capture time [10]- [13].…”

Section: Introductionmentioning

confidence: 99%

Real-Time Rendering of Point Clouds With Photorealistic Effects: A Survey

et al. 2022

View full text Add to dashboard Cite

Readily available RGB-D cameras in smart phones and improving 3D scanning technologies have made it possible to produce detailed point cloud and point-based models of real world objects even in real time. Rendering such models in high quality and at satisfactory frame rates is needed for realistic extended reality (XR) applications. This publication reviews real-time photorealistic point cloud rendering methods which directly ray trace or rasterize point cloud models, with an emphasis on ray tracing and realtime performance. We found that real-time direct point cloud ray tracing research has been focused on static non-animated content, and thus, open research possibilities include adapting modern dedicated ray tracing hardware for increased performance for animated and live captured scenes, and adding path tracing techniques to increase photorealistic effects in the rendering result. A categorization and discussion on the capabilities of state-of-the-art photorealistic point cloud rendering methods is presented by surveying both real-time and offline methods, which are assumed to become real-time capable with the advances in near-future hardware. Challenges and future trends are derived by comparing different rasterization and ray tracing methods as well as acceleration structures for point clouds in terms of produced rendering effects and speed.

show abstract

Capture, Reconstruction, and Representation of the Visual Real World for Virtual Reality

Cited by 22 publications

References 113 publications

A Simulation System for Scene Synthesis in Virtual Reality

A Simulation System for Scene Synthesis in Virtual Reality

Multimodality in VR: A Survey

Real-Time Rendering of Point Clouds With Photorealistic Effects: A Survey

Contact Info

Product

Resources

About