3D-RCNN: Instance-Level 3D Object Reconstruction via Render-and-Compare

Kundu, Anjan; Li, Yin; Rehg, James M.

doi:10.1109/cvpr.2018.00375

Cited by 322 publications

(297 citation statements)

References 38 publications

Supporting

Mentioning

296

Contrasting

Order By: Relevance

“…Several artificial intelligence (AI) research groups, including ours, have shown how neural network "inference models" can be built from a feedforward or recurrent network architecture trained to infer the underlying scene structure, rather than to recognize objects or classify object categories as in conventional DCNNs. In contrast to early analysis-by-synthesis algorithms, inference is fast, following a single bottom-up passes from the image or a small number of bottomup-top-down cycles, without the need for extensive iterative processing [24][25][26][27][28][29] . These models have been developed in an engineering setting, and are just beginning to be tested in machine vision problems; their correspondence with human perception or neural mechanisms is unexplored.…”

Section: Introductionmentioning

confidence: 99%

Efficient inverse graphics in biological face processing

Yildirim

Freiwald

Tenenbaum

2018

Preprint

View full text Add to dashboard Cite

Vision must not only recognize and localize objects, but perform richer inferences about the underlying causes in the world that give rise to sensory data. How the brain performs these inferences remains unknown: Theoretical proposals based on inverting generative models (or "analysis-by-synthesis") have a long history but their mechanistic implementations have typically been too slow to support online perception, and their mapping to neural circuits is unclear. Here we present a neurally plausible model for e ciently inverting generative models of images and test it as an account of one high-level visual capacity, the perception of faces. The model is based on a deep neural network that learns to invert a threedimensional (3D) face graphics program in a single fast feedforward pass. It explains both human behavioral data and multiple levels of neural processing in non-human primates, as well as a classic illusion, the "hollow face" e↵ect. The model fits qualitatively better than state-of-the-art computer vision models, and suggests an interpretable reverse-engineering account of how images are transformed into percepts in the ventral stream.

show abstract

Section: Introductionmentioning

confidence: 99%

Efficient inverse graphics in biological face processing

Yildirim

Freiwald

Tenenbaum

2018

Preprint

View full text Add to dashboard Cite

show abstract

“…Afterwards, estimated mesh information is used to recover the allocentric orientation [36] of the target object. Egocentric orientation can then be recovered and lifted to 6D by adopting different approaches from the literature [22,19].…”

Section: Methodsmentioning

confidence: 99%

“…[36] learned the best 3D surface keypoints for pose estimation, pursuing keypoints geometrically and semantically consistent from different viewpoint. Whereas target objects in other problems are in a categorical level [14,36,22,1], deformable and/or articulated, objects in 6D pose estimation are typically in an instance level and rigid. In this paper, we propose a deep architecture that recovers an object's 6D pose from the input of a single 2D RGB image via 3D reconstruction.…”

Section: Introductionmentioning

confidence: 99%

Accurate 6D Object Pose Estimation by Pose Conditioned Mesh Reconstruction

Castro

Armagan

Kim

2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Current 6D object pose methods consist of deep CNN models fully optimized for a single object but with its architecture standardized among objects with different shapes. In contrast to previous works, we explicitly exploit each object's distinct topological information i.e. 3D dense meshes in the pose estimation model, with an automated process and prior to any post-processing refinement stage. In order to achieve this, we propose a learning framework in which a Graph Convolutional Neural Network reconstructs a pose conditioned 3D mesh of the object. A robust estimation of the allocentric orientation is recovered by computing, in a differentiable manner, the Procrustes' alignment between the canonical and reconstructed dense 3D meshes. 6D egocentric pose is then lifted using additional mask and 2D centroid projection estimations. Our method is capable of self validating its pose estimation by measuring the quality of the reconstructed mesh, which is invaluable in real life applications. In our experiments on the LINEMOD, OCCLUSION and YCB-Video benchmarks, the proposed method outperforms state-of-the-arts.

show abstract

“…Different from [53,24] where render-and-compare losses are computed upon silhouettes and 2D depth maps, we compute dense render-and-compare losses L rac using IUV values between ground-truth IUV images and rendered ones (see Sec. 3.4).…”

Section: Dense Render-and-comparementioning

confidence: 99%

DenseRaC: Joint 3D Pose and Shape Estimation by Dense Render-and-Compare

Zhu

Tung

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

172

112

View full text Add to dashboard Cite

We present DenseRaC, a novel end-to-end framework for jointly estimating 3D human pose and body shape from a monocular RGB image. Our two-step framework takes the body pixel-to-surface correspondence map (i.e., IUV map) as proxy representation and then performs estimation of parameterized human pose and shape. Specifically, given an estimated IUV map, we develop a deep neural network optimizing 3D body reconstruction losses and further integrating a render-and-compare scheme to minimize differences between the input and the rendered output, i.e., dense body landmarks, body part masks, and adversarial priors. To boost learning, we further construct a large-scale synthetic dataset (MOCA) utilizing web-crawled Mocap sequences, 3D scans and animations. The generated data covers diversified camera views, human actions and body shapes, and is paired with full ground truth. Our model jointly learns to represent the 3D human body from hybrid datasets, mitigating the problem of unpaired training data. Our experiments show that DenseRaC obtains superior performance against state of the art on public benchmarks of various humanrelated tasks.

show abstract

3D-RCNN: Instance-Level 3D Object Reconstruction via Render-and-Compare

Cited by 322 publications

References 38 publications

Efficient inverse graphics in biological face processing

Efficient inverse graphics in biological face processing

Accurate 6D Object Pose Estimation by Pose Conditioned Mesh Reconstruction

DenseRaC: Joint 3D Pose and Shape Estimation by Dense Render-and-Compare

Contact Info

Product

Resources

About