GP2C: Geometric Projection Parameter Consensus for Joint 3D Pose and Focal Length Estimation in the Wild

Abstract

We present a joint 3D pose and focal length estimation approach for object categories in the wild. In contrast to previous methods that predict 3D poses independently of the focal length or assume a constant focal length, we explicitly estimate and integrate the focal length into the 3D pose estimation. For this purpose, we combine deep learning techniques and geometric algorithms in a two-stage approach: First, we estimate an initial focal length and establish 2D-3D correspondences from a single RGB image using a deep network. Second, we recover 3D poses and refine the focal length by minimizing the reprojection error of the predicted correspondences. In this way, we exploit the geometric prior given by the focal length for 3D pose estimation. This results in two advantages: First, we achieve significantly improved 3D translation and 3D pose accuracy compared to existing methods. Second, our approach finds a geometric consensus between the individual projection parameters, which is required for precise 2D-3D alignment. We evaluate our proposed approach on three challenging real-world datasets (Pix3D, Comp, and Stanford) with different object categories and significantly outperform the state-of-the-art by up to 20% absolute in multiple different metrics.

PoseContrast: Class-Agnostic Object Viewpoint Estimation in the Wild with Pose-Aware Contrastive Learning

“…While instancelevel 3D object pose estimation has long been studied in both robotic and vision communities [19,1,21,42,48,43,60,47,37,24], class-level pose estimation has developed more recently thanks to learning-based methods [45,52,51,34,23,55,14,13,56,66,50]. These methods can be roughly divided into two categories: direct pose estimation methods that regress 3D orientations directly [52,45,34,55,63], and keypoint-based methods that predict 2D locations of 3D keypoints [14,13,56,66,50]. However, annotating 3D poses for objects in the wild is a tedious process of searching best-matching CAD models and aligning them to images [59,58].…”

“…Object Detection on Pix3D. Results are given in AccD 0.5 as defined in [13]. We compare with two methods [55,13] that train a class-specific Mask R-CNN on COCO, then fine-tune on a subset of Pix3D containing the same classes as COCO.…”

“…Results are given in AccD 0.5 as defined in [13]. We compare with two methods [55,13] that train a class-specific Mask R-CNN on COCO, then fine-tune on a subset of Pix3D containing the same classes as COCO. In contrast, our agnostic Mask R-CNN is only trained on COCO and can generalize to classes not included in COCO.…”

ClusterVO: Clustering Moving Instances and Estimating Visual Odometry for Self and Surroundings

“…Detection and tracking in 3D space from video sequences is a relatively unexplored area due to the difficulty in the 6-DoF (six degrees of freedom) pose estimation. In order to accurately estimate 3D positions and poses, many methods [13,23] leverages a predefined object template or priors to jointly infer object depth and rotations. In ClusterVO, the combination of low-level geometric feature descriptors and semantic detections inferred simultaneously in the localization and mapping process can provide additional cues for efficient tracking and accurate object pose estimation.…”

Location Field Descriptors: Single Image 3D Model Retrieval in the Wild

“…Thus, it is important to know which pixels belong to an object and which pixels belong to the background or another object [7,8]. Recent works showed that deep learning techniques for instance segmentation [17] significantly increase the accuracy on this task [15,26,59]. However, until now location fields have only been used for 3D pose estimation, but not for 3D model retrieval or other tasks.…”