Vision-as-Inverse-Graphics: Obtaining a Rich 3D Explanation of a Scene from a Single Image

Romaszko, Łukasz; Williams, Christopher K. I.; Moreno, Pol; Kohli, Pushmeet

doi:10.1109/iccvw.2017.115

Cited by 28 publications

(22 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…EIG networks can be augmented with multiple scene layers in order to parse faces or other objects under occlusion 61, 62 . They be deployed in parallel or in series (using attention) to parse out multiple objects in a scene 24,[63][64][65][66] . They can even be extended to other modalities through which we perceive physical objects, such as touch, and can support flexible crossmodal transfer, allowing objects that have only been experienced in one modality (e.g., by sight) to be recognized in another (touch) 61 .…”

Section: Discussionmentioning

confidence: 99%

Efficient inverse graphics in biological face processing

Yildirim

Freiwald

Tenenbaum

2018

Preprint

View full text Add to dashboard Cite

Vision must not only recognize and localize objects, but perform richer inferences about the underlying causes in the world that give rise to sensory data. How the brain performs these inferences remains unknown: Theoretical proposals based on inverting generative models (or "analysis-by-synthesis") have a long history but their mechanistic implementations have typically been too slow to support online perception, and their mapping to neural circuits is unclear. Here we present a neurally plausible model for e ciently inverting generative models of images and test it as an account of one high-level visual capacity, the perception of faces. The model is based on a deep neural network that learns to invert a threedimensional (3D) face graphics program in a single fast feedforward pass. It explains both human behavioral data and multiple levels of neural processing in non-human primates, as well as a classic illusion, the "hollow face" e↵ect. The model fits qualitatively better than state-of-the-art computer vision models, and suggests an interpretable reverse-engineering account of how images are transformed into percepts in the ventral stream.

show abstract

Section: Discussionmentioning

confidence: 99%

Efficient inverse graphics in biological face processing

Yildirim

Freiwald

Tenenbaum

2018

Preprint

View full text Add to dashboard Cite

show abstract

“…from a Minecraft game). The work of IM2CAD [7] and [13] use such initialization methods in a realistic scenario and apply their models to real scenes.…”

Section: Learning Direct Optimizationmentioning

confidence: 99%

Learning Direct Optimization for scene understanding

Romaszko

Williams

Winn

2020

Pattern Recognition

Self Cite

View full text Add to dashboard Cite

We introduce a Learning Direct Optimization method for the refinement of a latent variable model that describes input image x. Our goal is to explain a single image x with a 3D computer graphics model having scene graph latent variables z (such as object appearance, camera position). Given a current estimate of z we can render a prediction of the image g(z), which can be compared to the image x. The standard way to proceed is then to measure the error E(x, g(z)) between the two, and use an optimizer to minimize the error. Our novel Learning Direct Optimization (LiDO) approach trains a Prediction Network to predict an update directly to correct z, rather than minimizing the error with respect to z. Experiments show that our LiDO method converges rapidly as it does not need to perform a search on the error landscape, produces better solutions, and is able to handle the mismatch between the data and the fitted scene model. We apply the LiDO to a realistic synthetic dataset, and show that the method transfers to work well with real images.

show abstract

“…Recent approaches employ CNNs to infer parameters of objects [26] or whole scenes [27] to aid procedural modeling. A similar trend is observed in graphics applications where CNNs are used to map input images or partial shapes to procedural model parameters [28]- [30].…”

Section: Inverse Procedural Modelingmentioning

confidence: 99%

CSGNet: Neural Shape Parser for Constructive Solid Geometry

Sharma

Goyal

Liu

et al. 2018

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

131

View full text Add to dashboard Cite

Constructive Solid Geometry (CSG) is a geometric modeling technique that defines complex shapes by recursively applying boolean operations on primitives such as spheres and cylinders. We present CSGNET, a deep network architecture that takes as input a 2D or 3D shape and outputs a CSG program that models it. Parsing shapes into CSG programs is desirable as it yields a compact and interpretable generative model. However, the task is challenging since the space of primitives and their combinations can be prohibitively large. CSGNET uses a convolutional encoder and recurrent decoder based on deep networks to map shapes to modeling instructions in a feed-forward manner and is significantly faster than bottom-up approaches. We investigate two architectures for this task -a vanilla encoder (CNN) -decoder (RNN) and another architecture that augments the encoder with an explicit memory module based on the program execution stack. The stack augmentation improves the reconstruction quality of the generated shape and learning efficiency. Our approach is also more effective as a shape primitive detector compared to a state-of-the-art object detector. Finally, we demonstrate CSGNET can be trained on novel datasets without program annotations through policy gradient techniques.

show abstract

Vision-as-Inverse-Graphics: Obtaining a Rich 3D Explanation of a Scene from a Single Image

Cited by 28 publications

References 13 publications

Efficient inverse graphics in biological face processing

Efficient inverse graphics in biological face processing

Learning Direct Optimization for scene understanding

CSGNet: Neural Shape Parser for Constructive Solid Geometry

Contact Info

Product

Resources

About