FrameNet: Learning Local Canonical Frames of 3D Surfaces From a Single RGB Image

Abstract

In this work, we introduce the novel problem of identifying dense canonical 3D coordinate frames from a single RGB image. We observe that each pixel in an image corresponds to a surface in the underlying 3D geometry, where a canonical frame can be identified as represented by three orthogonal axes, one along its normal direction and two in its tangent plane. We propose an algorithm to predict these axes from RGB. Our first insight is that canonical frames computed automatically with recently introduced direction field synthesis methods can provide training data for the task. Our second insight is that networks designed for surface normal prediction provide better results when trained jointly to predict canonical frames, and even better when trained to also predict 2D projections of canonical frames. We conjecture this is because projections of canonical tangent directions often align with local gradients in images, and because those directions are tightly linked to 3D canonical frames through projective geometry and orthogonality constraints. In our experiments, we find that our method predicts 3D canonical frames that can be used in applications ranging from surface normal estimation, feature matching, and augmented reality. Principal Directions RGB Images 3D Scans Local Principal Axes + Projected Tangent Principal Directions R e n d e r fo r S u p e r v is io n Normal Prediction Perspective Invariant Feature Matching Augmented Reality QuadriFlow CNN Camera Alignment

Pix2Shape: Towards Unsupervised Learning of 3D Scenes from Images Using a View-Based Representation

“…3D generation and reconstruction has been studied extensively in the computer vision and graphics communities (Saxena et al 2009;Chaudhuri et al 2011;Kalogerakis et al 2012;Chang et al 2015;Rezende et al 2016;Soltani et al 2017;Kulkarni et al 2015;Tulsiani et al 2016;Huang et al 2019;Jiang et al 2019). Most methods in the literature focus on recovering the 3D structure from 2D images by using explicit 3D supervision.…”

Automatic Dense Annotation for Monocular 3D Scene Understanding

“…Huang et al [28] propose FrameNet, a model to learn a canonical frame from a RGB image, where a canonical frame is represented by three orthogonal directions, one along the normal direction and two in the tangent plane. Dasgupta et al [29] propose a novel method called DeLay for room layout estimation from a single monocular RGB image that uses a CNN model to generate an initial belief map, which is then used by an optimization algorithm to predict the final room layout.…”

UprightNet: Geometry-Aware Camera Orientation Estimation From Single Images

“…Unfortunately, such method assumes the ground plane is always visible in the images, and only applies to vehicle-control use cases. In addition, there are recent work making use of local surface frame representation for a variety of 3D scene understanding tasks [20,21]. However, our method extends beyond these ideas by estimating both local and global aligned surface geometry from single images and use such correspondences to estimate camera orientation.…”

“…We find this choice leads to the best performance in our experiments. However, other surface frame representations could also be used [20,21].…”