Neural Inverse Rendering of an Indoor Scene From a Single Image

Sengupta, Shamik; Gu, Jun; Kim, Kihwan; Liu, Guilin; Jacobs, David W.; Kautz, Jan

doi:10.1109/iccv.2019.00869

Cited by 131 publications

(87 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Despite impressive results, artifacts remain especially around shadow boundaries and the relighting method fails beyond limited shadow motion. More recently, several learning based methods have been suggested to perform relighting in outdoor scenarios [48,47,31,34]. A simpler version of the relighting problem, is of integrating virtual objects into real scenes in an illumination-consistent manner, have been solved by using proxy geometry and user interaction [10,46,28,24].…”

Section: Related Workmentioning

confidence: 99%

Self-supervised Outdoor Scene Relighting

Meka

Elgharib

et al. 2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Outdoor scene relighting is a challenging problem that requires good understanding of the scene geometry, illumination and albedo. Current techniques are completely supervised, requiring high quality synthetic renderings to train a solution. Such renderings are synthesized using priors learned from limited data. In contrast, we propose a self-supervised approach for relighting. Our approach is trained only on corpora of images collected from the internet without any user-supervision. This virtually endless source of training data allows training a general relighting solution. Our approach first decomposes an image into its albedo, geometry and illumination. A novel relighting is then produced by modifying the illumination parameters. Our solution capture shadow using a dedicated shadow prediction map, and does not rely on accurate geometry estimation. We evaluate our technique subjectively and objectively using a new dataset with ground-truth relighting. Results show the ability of our technique to produce photo-realistic and physically plausible results, that generalizes to unseen scenes.

show abstract

Section: Related Workmentioning

confidence: 99%

Self-supervised Outdoor Scene Relighting

Meka

Elgharib

et al. 2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…The reconstruction can be performed by photographing light probes [11,12], labeling lights interactively [11], or be automated with an optimization process [13][14][15]. Deep neural networks can also learn relevant information from photographs, including lighting [16][17][18][19][20], geometry and albedo [21], or even SVBRDF [22][23][24][25]. Regardless of how the real scene is reconstructed, though, final compositing still comes to differential rendering [12,13,18,[26][27][28].…”

Section: Related Workmentioning

confidence: 99%

Neural compositing for real-time augmented reality rendering in low-frequency lighting environments

Shen

Hou

et al. 2021

Sci. China Inf. Sci.

View full text Add to dashboard Cite

We present neural compositing, a deep-learning based method for augmented reality rendering, which uses convolutional neural networks to composite rendered layers of a virtual object with a real photograph to emulate shadow and reflection effects. The method starts from estimating the lighting and roughness information from the photograph using neural networks, renders the virtual object with a virtual floor into color, shadow and reflection layers by applying the estimated lighting, and finally refines the reflection and shadow layers using neural networks and blends them with the color layer and input image to yield the output image. We assume low-frequency lighting environments and adopt PRT (precomputed radiance transfer) for layer rendering, which makes the whole pipeline differentiable and enables fast end-to-end network training with synthetic scenes. Working on a single photograph, our method can produce realistic reflections in a real scene with spatially-varying material and cast shadows on background objects with unknown geometry and material at real-time frame rates.

show abstract

“…In addition, image sequences over time are exploited to constrain the reflectance also within deep learning frameworks (Lettry et al 2018b;Li and Snavely 2018b). Finally, recent works on inverse scene rendering also aim at estimating scene-level reflectance maps (Sengupta et al 2019;Yu and Smith 2019;. Most of the intrinsic image decomposition algorithms represent shading as one unified component including all photometric effects.…”

Section: Related Workmentioning

confidence: 99%

“…Recently, supervised-based CNN methods use large-scale datasets (Shi et al 2017;Baslamisli et al 2018a;Li and Snavely 2018a;Sengupta et al 2019). Outdoor scenes are frequently influenced by strong shadow casts and varying lighting conditions.…”

Section: Related Workmentioning

confidence: 99%

ShadingNet: Image Intrinsics by Fine-Grained Shading Decomposition

Baslamisli

Das

et al. 2021

Int J Comput Vis

View full text Add to dashboard Cite

In general, intrinsic image decomposition algorithms interpret shading as one unified component including all photometric effects. As shading transitions are generally smoother than reflectance (albedo) changes, these methods may fail in distinguishing strong photometric effects from reflectance variations. Therefore, in this paper, we propose to decompose the shading component into direct (illumination) and indirect shading (ambient light and shadows) subcomponents. The aim is to distinguish strong photometric effects from reflectance variations. An end-to-end deep convolutional neural network (ShadingNet) is proposed that operates in a fine-to-coarse manner with a specialized fusion and refinement unit exploiting the fine-grained shading model. It is designed to learn specific reflectance cues separated from specific photometric effects to analyze the disentanglement capability. A large-scale dataset of scene-level synthetic images of outdoor natural environments is provided with fine-grained intrinsic image ground-truths. Large scale experiments show that our approach using fine-grained shading decompositions outperforms state-of-the-art algorithms utilizing unified shading on NED, MPI Sintel, GTA V, IIW, MIT Intrinsic Images, 3DRMS and SRD datasets.

show abstract

Neural Inverse Rendering of an Indoor Scene From a Single Image

Cited by 131 publications

References 47 publications

Self-supervised Outdoor Scene Relighting

Self-supervised Outdoor Scene Relighting

Neural compositing for real-time augmented reality rendering in low-frequency lighting environments

ShadingNet: Image Intrinsics by Fine-Grained Shading Decomposition

Contact Info

Product

Resources

About