Geometric Correspondence Fields: Learned Differentiable Rendering for 3D Pose Refinement in the Wild

Grabner, Alexander; Wang, Yaming; Zhang, Peizhao; Guo, Peihong; Xiao, Tong; Vajda, Péter; Roth, Peter M.; Lepetit, Vincent

doi:10.1007/978-3-030-58517-4_7

Cited by 6 publications

(3 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Though efficient, their formulation is built upon overlapped object regions across the reference image and target image, which is thus less robust against erroneous pose initialization. Grabner et al [48] proposed to refine the pose based on the correspondences, but their method is still limited to ideal scenarios.…”

Section: Related Workmentioning

confidence: 99%

RNNPose: 6-DoF Object Pose Estimation via Recurrent Correspondence Field Estimation and Pose Optimization

Xu,

Lin,

Zhang

et al. 2024

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

6-DoF object pose estimation from a monocular image is a challenging problem, where a post-refinement procedure is generally needed for high-precision estimation. In this paper, we propose a framework, dubbed RNNPose, based on a recurrent neural network (RNN) for object pose refinement, which is robust to erroneous initial poses and occlusions. During the recurrent iterations, object pose refinement is formulated as a non-linear least squares problem based on the estimated correspondence field (between a rendered image and the observed image). The problem is then solved by a differentiable Levenberg-Marquardt (LM) algorithm enabling end-to-end training. The correspondence field estimation and pose refinement are conducted alternately in each iteration to improve the object poses. Furthermore, to improve the robustness against occlusion, we introduce a consistency-check mechanism based on the learned descriptors of the 3D model and observed 2D images, which downweights the unreliable correspondences during pose optimization. We evaluate RNNPose on several public datasets, including LINEMOD, Occlusion-LINEMOD, YCB-Video and TLESS. We demonstrate state-of-the-art performance and strong robustness against severe clutter and occlusion in the scenes. Extensive experiments validate the effectiveness of our proposed method. Besides, the extended system based on RNNPose successfully generalizes to multi-instance scenarios and achieves top-tier performance on the TLESS dataset.

show abstract

Section: Related Workmentioning

confidence: 99%

RNNPose: 6-DoF Object Pose Estimation via Recurrent Correspondence Field Estimation and Pose Optimization

Xu,

Lin,

Zhang

et al. 2024

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…Generalizable pose estimators mostly require an object model either for shape embedding [69,42,12,43] or template matching [21,2,66,54,20,36,72] or renderingand-comparison [28,71,37,4,54,17]. To avoid using 3D models, recent works [70,40] utilize the advanced neural rendering techniques [35] to directly render from posed images for pose estimation.…”

Section: Generalizable Object Pose Estimatormentioning

confidence: 99%

Gen6D: Generalizable Model-Free 6-DoF Object Pose Estimation from RGB Images

Liu¹,

Wen²,

Peng³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this paper, we present a generalizable model-free 6-DoF object pose estimator called Gen6D. Existing generalizable pose estimators either need the high-quality object models or require additional depth maps or object masks in test time, which significantly limits their application scope. In contrast, our pose estimator only requires some posed images of the unseen object and is able to accurately predict poses of the object in arbitrary environments. Gen6D consists of an object detector, a viewpoint selector and a pose refiner, all of which do not require the 3D object model and can generalize to unseen objects. Experiments show that Gen6D achieves state-of-the-art results on two model-free datasets: the MOPED dataset and a new GenMOP dataset collected by us. In addition, on the LINEMOD dataset, Gen6D achieves competitive results compared with instance-specific pose estimators. Project page: https://liuyuan-pal.github.io/Gen6D/.

show abstract

“…This spawned several applications like self-supervision for monocular 3D object detection [5]. In [16], a differentiable renderer is employed -and even learned -to predict geometric correspondence fields to refine pose estimates of 3D objects. These approaches use rasterized rendering-schemes tightly integrated into a neural network.…”

Section: Related Workmentioning

confidence: 99%

In-Situ Joint Light and Medium Estimation for Underwater Color Restoration

Nakath

She

Song

et al. 2021

2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)

View full text Add to dashboard Cite

The majority of Earth's surface is situated in the deep sea and thus remains deprived of natural light. Such adverse underwater environments have to be explored with powerful camera-light systems. In order to restore the colors in images taken by such systems, we need to jointly estimate physically-meaningful optical parameters of the light as well as the water column. We thus propose an integrated in-situ estimation approach and a complementary surface texture recovery strategy, which also removes shadows as a by-product. As we operate in a scattering medium under inhomogeneous lighting conditions, the volumetric effects are difficult to capture in closed-form solutions. Hence, we leverage the latest progress in Monte Carlo-based differentiable ray tracing that becomes tractable through recent GPU RTX-hardware acceleration. Evaluations on synthetic data and in a water tank show that we can estimate physically meaningful parameters, which enables color restoration. The approaches could also be employed to other camera-light systems (AUV, robot, car, endoscope) operating either in the dark, in fog -or -underwater.

show abstract

Geometric Correspondence Fields: Learned Differentiable Rendering for 3D Pose Refinement in the Wild

Cited by 6 publications

References 42 publications

RNNPose: 6-DoF Object Pose Estimation via Recurrent Correspondence Field Estimation and Pose Optimization

RNNPose: 6-DoF Object Pose Estimation via Recurrent Correspondence Field Estimation and Pose Optimization

Gen6D: Generalizable Model-Free 6-DoF Object Pose Estimation from RGB Images

In-Situ Joint Light and Medium Estimation for Underwater Color Restoration

Contact Info

Product

Resources

About