Initial Assessment on the Use of State-of-the-Art Nerf Neural Network 3d Reconstruction for Heritage Documentation

Abstract

Abstract. In recent decades, photogrammetry has re-emerged as a viable solution for heritage documentation. Developments in various computer vision methods have helped photogrammetry to compete against the laser scanning technology, eventually becoming complementary solutions for the purpose of heritage recording. In the last few years, artificial intelligence (AI) has progressively entered various domains including 3D reconstruction. The Neural Radiance Fields (NeRF) method renders a 3D scene from a series of overlapping images, similar to photogrammetry. However, instead of relying on geometrical relations between the image and world spaces, it uses neural networks to recreate the so-called radiance fields. The result is a significantly faster method of recreating 3D scenes. While not designed to generate 3D models, simple computer graphics methods can be used to convert these recreated radiance fields into the familiar point cloud. In this paper, we implemented the Nerfacto architecture to recreate two instances of heritage objects and then compared them to traditional photogrammetric multi-view stereo (MVS). While the initial hypothesis posits that NeRF is not yet capable to reach the level of accuracy and density achieved by MVS as can be observed in the results, NeRF nevertheless shows a great potential due to its fractionally faster processing speed.

Automatic Removal of Non-Architectural Elements in 3D Models of Historic Buildings with Language Embedded Radiance Fields

“…Nerfstudio is an open source NeRF development software which provides viewers and standard components of various NeRF architectures. In reimplementing multiple architectures (MipNeRF-360 [25], NeRF- [26], Instant-NGP [27], NeRF-W [28], and Ref-NeRF [29]) in a standardized, modular fashion, the authors were able to then use various components to create a model, Nerfacto, that is both fast and relatively accurate; close or surpassing the state of the art for NeRFs in most quality metrics while training substantially faster [30,31]. There are multiple important differences between a base NeRF and the Nerfacto approach in the optimization of camera positions, ray sampling techniques, scene registration, hashgrid encoding, and the generation of normals.…”

Comparative Assessment of Neural Radiance Fields and Photogrammetry in Digital Heritage: Impact of Varying Image Conditions on 3D Reconstruction

“…However, the interest in applying NeRFs compared to other well-established technologies like photogrammetry has still not yet been recognized to its full potential. Even though the starting point is still a series of overlapping images, NeRFs use neural networks to create so-called radiance fields instead of relying on the reconstruction of geometrical relations between an image and the 3D world space [17]. The output, provided in the form of a neural rendering, can be turned into common, consolidated 3D objects as point clouds or meshes, e.g., via the marching cubes algorithm [18].…”

“…Interest in the application of NeRFs to the cultural heritage sector is on the rise. Murtiyoso and Grussenmeyer [17] empirically demonstrated that NeRFs exhibit accelerated processing times in contrast to conventional MVS methods, but they noted a trade-off in geometric precision and the level of detail documentation. To assess the geometric quality of outputs, two heritage objects are subjected to evaluation, considering the mean error and standard deviation values across the resulting point clouds.…”

Comparative Assessment of NeRF and Photogrammetry in Digital Heritage: Impact of Varying Image Conditions on 3D Reconstruction

“…However, the interest in applying NeRFs compared to other well-established technologies like photogrammetry is still not yet recognized to its full potential. Even though the starting point is still a series of overlapping images, NeRF use neural networks to create the so-called radiance fields, instead of relying on the reconstruction of geometrical relations between image and 3D world space [17]. The output, provided in the form of a neural rendering, can be turned into common, consolidated 3D objects as point clouds or meshes, e.g., via the marching cubes algorithm [18].…”