Pixel-Pair Occlusion Relationship Map (P2ORM): Formulation, Inference and Application

Abstract

We formalize concepts around geometric occlusion in 2D images (i.e., ignoring semantics), and propose a novel unified formulation of both occlusion boundaries and occlusion orientations via a pixel-pair occlusion relation. The former provides a way to generate large-scale accurate occlusion datasets while, based on the latter, we propose a novel method for task-independent pixel-level occlusion relationship estimation from single images. Experiments on a variety of datasets demonstrate that our method outperforms existing ones on this task. To further illustrate the value of our formulation, we also propose a new depth map refinement method that consistently improve the performance of stateof-the-art monocular depth estimation methods.

FusionNet: Coarse-to-Fine Extrinsic Calibration Network of LiDAR and Camera with Hierarchical Point-pixel Fusion

“…III-A, the surface-pair occlusion relationship in 3D can be represented as the pixel-pair occlusion relationship in 2D. For each valid image pixel pair (q i , q j ), neural network P2ORNet [15] classifies the occlusion relationship status among three possible three status: q i occludes q j , q j occludes q i and no occlusion between (q i and q j ). The estimated image occlusion edge lies on the image region where P2ORNet predicts that occlusion exists between pixel pairs.…”

“…3) Network Training: The P2ORNet is trained with a class-balanced cross-entropy loss [15], taking into account the low-class frequency of pixels on image occlusion edges (cf. Sec.…”

“…The full optimization step is summarized in the The model of P2ORNet used for all the following tests is trained on the virtual KITTI2 (vKITTI2) [34] dataset. We use the depth images of vKITTI2 and the tools provided by [15] to generate the occlusion relationship ground truths. In order to get rid of the trivial but noisy occlusion image features, e.g., features from leaves on the trees, occlusion labels belonging to these categories of objects are masked out in the training dataset.…”

“…To alleviate the aforementioned limitations, we propose a novel target-free extrinsic calibration method which uses the unified 2D-3D occlusion feature to achieve robust estimation in various scenarios. For feature extraction on camera images, our previous work [15] introduces the concept of pixel-pair occlusion relationship and allows us to extract the image edges points with spatial occlusion information. In this work, we exploit the 3D occlusion edge extraction on LiDAR point clouds following the occlusion relationship arXiv:2311.02413v1 [cs.CV] 4 Nov 2023 definition, and develop a novel calibration approach using the extracted 2D-3D point pairs.…”

PlaneTR: Structure-Guided Transformers for 3D Plane Recovery

“…Structures have been successfully used in some learningbased methods [35,12,27,29,26]. Wang et al [35] propose to learn a Manhattan Label Map from the input RGB image and its corresponding Manhattan line map for normal esti- mation.…”