Explorations for real-time point cloud rendering of natural scenes in virtual reality

Bonatto, Daniele; Rogge, Segolene; Schenkel, Arnaud; Ercek, Rudy; Lafruit, Gauthier

doi:10.1109/ic3d.2016.7823453

“…Compression and decompression will potentially result in noisy point clouds as well (Cao et al, 2019). Besides, real-time rendering in VR games is another source of noise (Bonatto et al, 2016). Although noise is a common corruption pattern for both 2D and 3D data, the manip-…”

Section: Noise Corruption Patternsmentioning

confidence: 99%

Benchmarking Robustness of 3D Point Cloud Recognition Against Common Corruptions

Sun¹,

Zhang²,

Kailkhura³

et al. 2022

Preprint

View full text Add to dashboard Cite

Deep neural networks on 3D point cloud data have been widely used in the real world, especially in safety-critical applications. However, their robustness against corruptions is less studied. In this paper, we present ModelNet40-C, the first comprehensive benchmark on 3D point cloud corruption robustness, consisting of 15 common and realistic corruptions. Our evaluation shows a significant gap between the performances on ModelNet40 and ModelNet40-C for state-of-the-art (SOTA) models. To reduce the gap, we propose a simple but effective method by combining PointCutMix-R and TENT after evaluating a wide range of augmentation and testtime adaptation strategies. We identify a number of critical insights for future studies on corruption robustness in point cloud recognition. For instance, we unveil that Transformer-based architectures with proper training recipes achieve the strongest robustness. We hope our in-depth analysis will motivate the development of robust training strategies or architecture designs in the 3D point cloud domain. Our codebase and dataset are included in https://github. com/jiachens/ModelNet40-C.

show abstract

“…While it is possible to directly render the point clouds [34,9], doing so has two key disadvantages. First, pure point clouds are just not visually appealing.…”

Section: Point Cloud Reconstruction For Mobile Vrmentioning

confidence: 99%

Digital reconstruction of Elmina Castle for mobile virtual reality via point-based detail transfer

Ye¹,

Ting²,

Jarvis³

et al. 2022

ei

1

0

View full text Add to dashboard Cite

Reconstructing 3D models from large, dense point clouds is critical to enable Virtual Reality (VR) as a platform for entertainment, education, and heritage preservation. Existing 3D reconstruction systems inevitably make trade-offs between three conflicting goals: the efficiency of reconstruction (e.g., time and memory requirements), the visual quality of the constructed scene, and the rendering speed on the VR device. This paper proposes a reconstruction system that simultaneously meets all three goals. The key idea is to avoid the resource-demanding process of reconstructing a high-polygon mesh altogether. Instead, we propose to directly transfer details from the original point cloud to a low polygon mesh, which significantly reduces the reconstruction time and cost, preserves the scene details, and enables real-time rendering on mobile VR devices.While our technique is general, we demonstrate it in reconstructing cultural heritage sites. We for the first time digitally reconstruct the Elmina Castle, a UNESCO world heritage site at Ghana, from billions of laser-scanned points. The reconstruction process executes on low-end desktop systems without requiring high processing power, making it accessible to the broad community. The reconstructed scenes render on Oculus Go in 60 FPS, providing a real-time VR experience with high visual quality.

show abstract

“…While it is possible to directly render the point clouds [6,27], doing so has two key disadvantages. First, pure point clouds are just not visually appealing.…”

Section: Point Cloud Reconstruction For Mobile Vrmentioning

confidence: 99%

Digital Reconstruction of Elmina Castle for Mobile Virtual Reality via Point-based Detail Transfer

Ye¹,

Wu²,

Jarvis³

et al. 2020

Preprint

1

0

View full text Add to dashboard Cite

Reconstructing 3D models from large, dense point clouds is critical to enable Virtual Reality (VR) as a platform for entertainment, education, and heritage preservation. Existing 3D reconstruction systems inevitably make trade-offs between three conflicting goals: the efficiency of reconstruction (e.g., time and memory requirements), the visual quality of the constructed scene, and the rendering speed on the VR device. This paper proposes a reconstruction system that simultaneously meets all three goals. The key idea is to avoid the resource-demanding process of reconstructing a high-polygon mesh altogether. Instead, we propose to directly transfer details from the original point cloud to a low polygon mesh, which significantly reduces the reconstruction time and cost, preserves the scene details, and enables real-time rendering on mobile VR devices.While our technique is general, we demonstrate it in reconstructing cultural heritage sites. We for the first time digitally reconstruct the Elmina Castle, a UNESCO world heritage site at Ghana, from billions of laser-scanned points. The reconstruction process executes on low-end desktop systems without requiring high processing power, making it accessible to the broad community. The reconstructed scenes render on Oculus Go in 60 FPS, providing a real-time VR experience with high visual quality. Our project is part of the Digital Elmina effort (http://digitalelmina.org/) between

show abstract

Explorations for real-time point cloud rendering of natural scenes in virtual reality

Cited by 11 publications

References 9 publications

Benchmarking Robustness of 3D Point Cloud Recognition Against Common Corruptions

Benchmarking Robustness of 3D Point Cloud Recognition Against Common Corruptions

Digital reconstruction of Elmina Castle for mobile virtual reality via point-based detail transfer

Digital Reconstruction of Elmina Castle for Mobile Virtual Reality via Point-based Detail Transfer

Contact Info

Product

Resources

About