6D Camera Relocalization in Ambiguous Scenes via Continuous Multimodal Inference

Bui, Mai; Birdal, Tolga; Deng, Haowen; Albarqouni, Shadi; Guibas, Leonidas J.; Ilić, Slobodan; Navab, Nassir

doi:10.1007/978-3-030-58523-5_9

Cited by 21 publications

(17 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Figure 6 shows that the flexibility of EPro-PnP allows predicting multimodal distributions with strong expressive power, successfully capturing the orientation ambiguity without discrete multi-bin classification [33,47] or complicated mixture model [7]. Owing to the ability to model orientation ambiguity, EPro-PnP outperforms other competitors by a wide margin in terms of the AOE metric in Table 4.…”

Section: Qualitative Analysismentioning

confidence: 98%

“…This inspired works such as DSAC [4], a smooth RANSAC with a finite hypothesis pool. Meanwhile, simple parametric distributions (e.g., normal distribution) are often used in predicting continuous variables [13,18,22,25,26,51], and mixture distributions can be employed to further capture ambiguity [3,5,31], e.g., ambiguous 6DoF pose [7]. In this paper, we propose yet a unique contribution: backpropagating a complicated continuous distribution derived from a nested optimization layer (the PnP layer), essentially making the continuous counterpart of Softmax tractable.…”

Section: End-to-end Correspondence Learningmentioning

confidence: 99%

See 1 more Smart Citation

EPro-PnP: Generalized End-to-End Probabilistic Perspective-n-Points for Monocular Object Pose Estimation

Chen¹,

Wang²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Locating 3D objects from a single RGB image via Perspective-n-Points (PnP) is a long-standing problem in computer vision. Driven by end-to-end deep learning, recent studies suggest interpreting PnP as a differentiable layer, so that 2D-3D point correspondences can be partly learned by backpropagating the gradient w.r.t. object pose. Yet, learning the entire set of unrestricted 2D-3D points from scratch fails to converge with existing approaches, since the deterministic pose is inherently non-differentiable. In this paper, we propose the EPro-PnP, a probabilistic PnP layer for general end-to-end pose estimation, which outputs a distribution of pose on the SE(3) manifold, essentially bringing categorical Softmax to the continuous domain. The 2D-3D coordinates and corresponding weights are treated as intermediate variables learned by minimizing the KL divergence between the predicted and target pose distribution. The underlying principle unifies the existing approaches and resembles the attention mechanism. EPro-PnP significantly outperforms competitive baselines, closing the gap between PnP-based method and the taskspecific leaders on the LineMOD 6DoF pose estimation and nuScenes 3D object detection benchmarks. 3 * Part of work done during an internship at Alibaba Group.

show abstract

Section: Qualitative Analysismentioning

confidence: 98%

Section: End-to-end Correspondence Learningmentioning

confidence: 99%

EPro-PnP: Generalized End-to-End Probabilistic Perspective-n-Points for Monocular Object Pose Estimation

Chen¹,

Wang²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Using a state-of-the-art matching architecture would likely improve the results of NRE but we left this as future work. End-to-end camera pose estimation methods [9,10,12,22,23,28,47,52] learn jointly all the parameters of the camera pose estimator by backpropagating through it. Different architectures of camera pose estimators have been proposed in the literature.…”

Section: Related Workmentioning

confidence: 99%

Neural Reprojection Error: Merging Feature Learning and Camera Pose Estimation

Germain,

Lepetit,

Bourmaud

2021

Preprint

View full text Add to dashboard Cite

Absolute camera pose estimation is usually addressed by sequentially solving two distinct subproblems: First a feature matching problem that seeks to establish putative 2D-3D correspondences, and then a Perspective-n-Point problem that minimizes, w.r.t. the camera pose, the sum of socalled Reprojection Errors (RE). We argue that generating putative 2D-3D correspondences 1) leads to an important loss of information that needs to be compensated as far as possible, within RE, through the choice of a robust loss and the tuning of its hyperparameters and 2) may lead to an RE that conveys erroneous data to the pose estimator. In this paper, we introduce the Neural Reprojection Error (NRE) as a substitute for RE. NRE allows to rethink the camera pose estimation problem by merging it with the feature learning problem, hence leveraging richer information than 2D-3D correspondences and eliminating the need for choosing a robust loss and its hyperparameters. Thus NRE can be used as training loss to learn image descriptors tailored for pose estimation. We also propose a coarse-to-fine optimization method able to very efficiently minimize a sum of NRE terms w.r.t. the camera pose. We experimentally demonstrate that NRE is a good substitute for RE as it significantly improves both the robustness and the accuracy of the camera pose estimate while being computationally and memory highly efficient. From a broader point of view, we believe this new way of merging deep learning and 3D geometry may be useful in other computer vision applications. Source code and model weights will be made available at hugogermain.com/nre.

show abstract

“…First, in most applications of interest, dynamics are attributed to rigid object motion [9,42]. This notion has been extensively exploited in robotics [8,13,7,6] and holds especially for vehicles in autonomous driving. Predicting unconstrained per-point flow may lead to non-viable results, e.g.…”

Section: Introductionmentioning

confidence: 99%

Weakly Supervised Learning of Rigid 3D Scene Flow

Gojčič

Litany

Guibas

et al. 2021

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

View full text Add to dashboard Cite

We propose a data-driven scene flow estimation algorithm exploiting the observation that many 3D scenes can be explained by a collection of agents moving as rigid bodies. At the core of our method lies a deep architecture able to reason at the object-level by considering 3D scene flow in conjunction with other 3D tasks. This object level abstraction enables us to relax the requirement for dense scene flow supervision with simpler binary background segmentation mask and ego-motion annotations. Our mild supervision requirements make our method well suited for recently released massive data collections for autonomous driving, which do not contain dense scene flow annotations. As output, our model provides low-level cues like pointwise flow and higher-level cues such as holistic scene understanding at the level of rigid objects. We further propose a test-time optimization refining the predicted rigid scene flow. We showcase the effectiveness and generalization capacity of our method on four different autonomous driving datasets. We release our source code and pre-trained models under github.com/zgojcic/Rigid3DSceneFlow.

show abstract

6D Camera Relocalization in Ambiguous Scenes via Continuous Multimodal Inference

Cited by 21 publications

References 55 publications

EPro-PnP: Generalized End-to-End Probabilistic Perspective-n-Points for Monocular Object Pose Estimation

EPro-PnP: Generalized End-to-End Probabilistic Perspective-n-Points for Monocular Object Pose Estimation

Neural Reprojection Error: Merging Feature Learning and Camera Pose Estimation

Weakly Supervised Learning of Rigid 3D Scene Flow

Contact Info

Product

Resources

About