Loc-NeRF: Monte Carlo Localization using Neural Radiance Fields

Abstract

We present Loc-NeRF, a real-time vision-based robot localization approach that combines Monte Carlo localization and Neural Radiance Fields (NeRF). Our system uses a pre-trained NeRF model as the map of an environment and can localize itself in real-time using an RGB camera as the only exteroceptive sensor onboard the robot. While neural radiance fields have seen significant applications for visual rendering in computer vision and graphics, they have found limited use in robotics. Existing approaches for NeRF-based localization require both a good initial pose guess and significant computation, making them impractical for real-time robotics applications. By using Monte Carlo localization as a workhorse to estimate poses using a NeRF map model, Loc-NeRF is able to perform localization faster than the state of the art and without relying on an initial pose estimate. In addition to testing on synthetic data, we also run our system using real data collected by a Clearpath Jackal UGV and demonstrate for the first time the ability to perform real-time global localization with neural radiance fields. We make our code publicly available at https://github.com/MIT-SPARK/Loc-NeRF.

PNeRFLoc: Visual Localization with Point-Based Neural Radiance Fields

“…However, the training process remains lengthy, as DFNet still needs to train NeRF, pose regression, and feature extraction networks separately. Maggio et al proposed a Monte Carlo localization method called Loc-NeRF (Maggio et al 2022), where Loc-NeRF continuously samples candidate poses under the initial pose and uses NeRF to render novel views to find the correct pose direction. However, Loc-NeRF is unstable and still requires an initial camera pose.…”

“…Moreover, from the adaptation module, we can also learn a score for each dense feature for better feature matching and initial localization. Second, the renderingbased optimization may be easily stuck in the local minimum (Maggio et al 2022) due to the backpropagation through the networks and also time-consuming. To improve the neural rendering-based optimization with point-based representation, we further propose a novel efficient rendering-based optimization framework by aligning the rendered image with the query image and minimizing the warping loss function.…”

Renderable Neural Radiance Map for Visual Navigation

“…NeRF predicts the RGB color and density of a point in a scene so that an image from an arbitrary viewpoint can be rendered. This property enables pose estimation [1,30,31,44] based on the photometric loss between the observed image and the rendered image, or manipulation of tricky objects [5,12,14,25,29]. A pretrained NeRF can also work as a virtual simulator, in which a robot can plan its trajectory [1] or can be used to train an action policy for the real-world [6].…”

“…There have been a number of studies that leverage neural radiance fields (NeRF) for various applications other than novel view synthesis. The robotic applications of NeRF are also now beginning to emerge [1,15,31,44,55]. However, many of the approaches require pretrained neural radiance fields about a specific scene and are not generalizable to various scenes.…”

A Study of Latent Tuberculosis Infection And its Influencing Factors in Diabetic Patients in Nanshan District, Shenzhen, China

“…Scene coordinate regression directly predicts absolute 3D coordinates of image pixels and uses scene structure explicitly to improve accuracy. More recently, many efforts [28] [38][22] [17] have been made to use implicit neural representation to replace explicit 3D models in localization pipeline. Different from the commonly used discrete 3D models like point cloud and voxel grid, NeRF is one of the implicit 3D representations inferred from a sparse set of posed images, which models geometry and visual information in continuous 3D space.…”