Beyond Controlled Environments: 3D Camera Re-localization in Changing Indoor Scenes

Wald, Johanna; Sattler, Torsten; Golodetz, Stuart; Cavallari, Tommaso; Tombari, Federico

doi:10.1007/978-3-030-58571-6_28

Cited by 35 publications

(31 citation statements)

References 96 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We compare our sequence-based results with generalized cameras, as implemented by Wald et al [87], and stateof-the-art, published single-image methods chosen based on the top-performing methods from visuallocalization.net. For the generalized cameras, the components D and E are replaced by a generalized PnP solver [44], while keeping components A, B and C as before.…”

Section: Experimental Evaluationmentioning

confidence: 99%

Using Image Sequences for Long-Term Visual Localization

Stenborg

Sattler

Hammarstrand

2020

2020 International Conference on 3D Vision (3DV)

Self Cite

View full text Add to dashboard Cite

Estimating the pose of a camera in a known scene, i.e., visual localization, is a core task for applications such as self-driving cars. In many scenarios, image sequences are available and existing work on combining single-image localization with odometry offers to unlock their potential for improving localization performance. Still, the largest part of the literature focuses on single-image localization and ignores the availability of sequence data. The goal of this paper is to demonstrate the potential of image sequences in challenging scenarios, e.g., under day-night or seasonal changes. Combining ideas from the literature, we describe a sequence-based localization pipeline that combines odometry with both a coarse and a fine localization module. Experiments on long-term localization datasets show that combining single-image global localization against a prebuilt map with a visual odometry / SLAM pipeline improves performance to a level where the extended CMU Seasons dataset can be considered solved. We show that SIFT features can perform on par with modern state-of-the-art features in our framework, despite being much weaker and a magnitude faster to compute. Our code is publicly available at github.com/rulllars.

show abstract

Section: Experimental Evaluationmentioning

confidence: 99%

Using Image Sequences for Long-Term Visual Localization

Stenborg

Sattler

Hammarstrand

2020

2020 International Conference on 3D Vision (3DV)

Self Cite

View full text Add to dashboard Cite

show abstract

“…3D Scene Retrieval Visual retrieval has a long history in computer vision, and is often embedded in other tasks such as object detection, object or scene alignment or camera pose estimation (Gálvez-López and Tardós 2011; Torii et al 2015;Glocker et al 2015;Anoosheh et al 2019;Arandjelović et al 2016;Deng et al 2016;Lu et al 2016b). Retrieving a source given some query data becomes most challenging when they do not share the same domain (Dahnert et al 2019;Abdul-Rashid et al 2018Avetisyan et al 2019Avetisyan et al , 2020 including natural scene changes (Wald et al 2020b(Wald et al , 2019. An extensive literature exists on retrieval of CAD models given an image (Izadinia et al 2017;Sun et al 2018) or a 3D scene (Avetisyan et al 2019(Avetisyan et al , 2020.…”

Section: D Object Context and Scene Layoutmentioning

confidence: 99%

“…Besides the full scene graphs, a smaller dataset that features a subset of objects and predicates is also made available. 3 Additionally to 3DSSG that has been released in Wald et al (2020a) we also provide 2D semantic scene graphs for the camera re-localization benchmark, RIO10 (Wald et al 2020b). In summary, we offer (a) scene graphs, (b) RGB-D sequences with camera poses and intrinsics, (c) textured 3D models with point coordinates and surface normals…”

Section: D Semantic Scene Graphsmentioning

confidence: 99%

“…The rendering pipeline and 2D graphs for the camera re-localization benchmark, RIO10 (Wald et al 2020b) will be made publicly available 4 . Compared to some other 2D scene graph datasets (Krishna et al 2017) our data provides depth images and semantic instance masks -additionally to bounding boxes -similarly to the much smaller NYUv2 (Nathan Silberman Derek Hoiem and Fergus 2012).…”

Section: D Scene Graphsmentioning

confidence: 99%

“…b) We further publish scene graphs for the localization benchmark RIO10 complementary to the large scale 3D scene graph dataset, 3DSSG (Wald et al 2020a) 2 . The datasets are an extension of RIO10 Wald et al 2020b and3RScan (Wald et al 2019) and include graph annotations in form of relationships, instance attributes and class label hierarchies for each instance. c) We finally show the effectiveness of such graphs in a retrieval application.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Learning 3D Semantic Scene Graphs with Instance Embeddings

2022

Self Cite

View full text Add to dashboard Cite

A 3D scene is more than the geometry and classes of the objects it comprises. An essential aspect beyond object-level perception is the scene context, described as a dense semantic network of interconnected nodes. Scene graphs have become a common representation to encode the semantic richness of images, where nodes in the graph are object entities connected by edges, so-called relationships. Such graphs have been shown to be useful in achieving state-of-the-art performance in image captioning, visual question answering and image generation or editing. While scene graph prediction methods so far focused on images, we propose instead a novel neural network architecture for 3D data, where the aim is to learn to regress semantic graphs from a given 3D scene. With this work, we go beyond object-level perception, by exploring relations between object entities. Our method learns instance embeddings alongside a scene segmentation and is able to predict semantics for object nodes and edges. We leverage 3DSSG, a large scale dataset based on 3RScan that features scene graphs of changing 3D scenes. Finally, we show the effectiveness of graphs as an intermediate representation on a retrieval task.

show abstract

Map-Free Visual Relocalization: Metric Pose Relative to a Single Image

Arnold

Wynn²,

Vicente³

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Can we relocalize in a scene represented by a single reference image? Standard visual relocalization requires hundreds of images and scale calibration to build a scene-specific 3D map. In contrast, we propose Map-free Relocalization, i.e., using only one photo of a scene to enable instant, metric scaled relocalization. Existing datasets are not suitable to benchmark map-free relocalization, due to their focus on large scenes or their limited variability. Thus, we have constructed a new dataset of 655 small places of interest, such as sculptures, murals and fountains, collected worldwide. Each place comes with a reference image to serve as a relocalization anchor, and dozens of query images with known, metric camera poses. The dataset features changing conditions, stark viewpoint changes, high variability across places, and queries with low to no visual overlap with the reference image. We identify two viable families of existing methods to provide baseline results: relative pose regression, and feature matching combined with single-image depth prediction. While these methods show reasonable performance on some favorable scenes in our dataset, map-free relocalization proves to be a challenge that requires new, innovative solutions.

show abstract

Beyond Controlled Environments: 3D Camera Re-localization in Changing Indoor Scenes

Cited by 35 publications

References 96 publications

Using Image Sequences for Long-Term Visual Localization

Using Image Sequences for Long-Term Visual Localization

Learning 3D Semantic Scene Graphs with Instance Embeddings

Map-Free Visual Relocalization: Metric Pose Relative to a Single Image

Contact Info

Product

Resources

About