Cuboids Revisited: Learning Robust 3D Shape Fitting to Single RGB Images

Abstract

Humans perceive and construct the surrounding world as an arrangement of simple parametric models. In particular, man-made environments commonly consist of volumetric primitives such as cuboids or cylinders. Inferring these primitives is an important step to attain high-level, abstract scene descriptions. Previous approaches directly estimate shape parameters from a 2D or 3D input, and are only able to reproduce simple objects, yet unable to accurately parse more complex 3D scenes. In contrast, we propose a robust estimator for primitive fitting, which can meaningfully abstract real-world environments using cuboids. A RANSAC estimator guided by a neural network fits these primitives to 3D features, such as a depth map. We condition the network on previously detected parts of the scene, thus parsing it one-by-one. To obtain 3D features from a single RGB image, we additionally optimise a feature extraction CNN in an end-to-end manner. However, naively minimising pointto-primitive distances leads to large or spurious cuboids occluding parts of the scene behind. We thus propose an occlusion-aware distance metric correctly handling opaque scenes. The proposed algorithm does not require labourintensive labels, such as cuboid annotations, for training. Results on the challenging NYU Depth v2 dataset demonstrate that the proposed algorithm successfully abstracts cluttered real-world 3D scene layouts.

Primitive-Based 3D Human-Object Interaction Modelling and Programming

“…Considering these problems, primitive-based models are proposed (Yao et al 2021;Kluger et al 2021;Yavartanoo et al 2021;Paschalidou et al 2021;Han et al 2021;He et al 2021;Yao et al 2022Yao et al , 2023b. Most use rendered 3D models as input (Yao et al 2021;Yavartanoo et al 2021;Paschalidou et al 2021;Han et al 2021), but few use realis-tic photos (Kluger et al 2021;He et al 2021;Yao et al 2022Yao et al , 2023b, among which (Yao et al 2022(Yao et al , 2023b are most similar to ours. However, they only focus on one animal in one image case, and the input images need to be carefully curated or pre-processed.…”