Learning Intrinsic Image Decomposition from Watching the World

Li, Zhengqi; Snavely, Noah

doi:10.1109/cvpr.2018.00942

Cited by 146 publications

(154 citation statements)

References 38 publications

Supporting

Mentioning

154

Contrasting

Order By: Relevance

“…Third, our network could benefit from losses used in training intrinsic image decomposition networks. For example, if we added the timelapse dataset of [33] to our training, we could incorporate their reflectance consistency loss to improve our albedo map estimates. Our code, trained model and inverse rendering benchmark data is available at <URL removed for review>.…”

Section: Discussionmentioning

confidence: 99%

“…Recent work either uses synthetic training data and supervised learning [7,12,20,30,37] or self-supervision/unsupervised learning. Very recently, Li et al [33] used uncontrolled time-lapse images allowing them to combine an image reconstruction loss with reflectance consistency between frames. This work was further extended using photorealistic, synthetic training data [32].…”

Section: Related Workmentioning

confidence: 99%

“…We choose our crops to maximise the number of pixels with defined depth values. Where possible, we crop multiple images from each image, achieving augmen-Images Li [33] (R)…”

Section: Datasetsmentioning

confidence: 99%

“…Nestmeyer [38] (R) Ours (R) Li [33] (S) Nestmeyer [38] (S) Ours (S) Figure 5: Qualitative results for IIW. Second column to forth column are reflectance predictions from [33], [38] and ours. The last three columns are corresponding shading predictions.…”

Section: Datasetsmentioning

confidence: 99%

See 3 more Smart Citations

InverseRenderNet: Learning Single Image Inverse Rendering

Smith

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

133

117

View full text Add to dashboard Cite

InputDiffuse albedo Illumination NM prediction NM from MVS Frontal shading Shading Figure 1: From a single image (col. 1), we estimate albedo and normal maps and illumination (col. 2-4); comparison multiview stereo result from several hundred images (col. 5); re-rendering of our shape with frontal/estimated lighting (col. 6-7). AbstractWe show how to train a fully convolutional neural network to perform inverse rendering from a single, uncontrolled image. The network takes an RGB image as input, regresses albedo and normal maps from which we compute lighting coefficients. Our network is trained using large uncontrolled image collections without ground truth. By incorporating a differentiable renderer, our network can learn from self-supervision. Since the problem is ill-posed we introduce additional supervision: 1. We learn a statistical natural illumination prior, 2. Our key insight is to perform offline multiview stereo (MVS) on images containing rich illumination variation. From the MVS pose and depth maps, we can cross project between overlapping views such that Siamese training can be used to ensure consistent estimation of photometric invariants. MVS depth also provides direct coarse supervision for normal map estimation. We believe this is the first attempt to use MVS supervision for learning inverse rendering.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

“…We choose our crops to maximise the number of pixels with defined depth values. Where possible, we crop multiple images from each image, achieving augmen-Images Li [33] (R)…”

Section: Datasetsmentioning

confidence: 99%

Section: Datasetsmentioning

confidence: 99%

See 2 more Smart Citations

InverseRenderNet: Learning Single Image Inverse Rendering

Smith

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

133

117

View full text Add to dashboard Cite

show abstract

“…It is also possible to extract illumination, material and geometry information from time-lapse videos as shown in previous methods [23,29,17]. Most recently, Li and Snavely [20] proposed to learn singleview intrinsic image decomposition from time-lapse videos in the wild without ground truth data. We draw inspirations from this line of research and propose to learn a generative model for the time-lapse video synthesis.…”

Section: Related Workmentioning

confidence: 99%

End-To-End Time-Lapse Video Synthesis From a Single Outdoor Image

Nam

Chai

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Time-lapse videos usually contain visually appealing content but are often difficult and costly to create. In this paper, we present an end-to-end solution to synthesize a time-lapse video from a single outdoor image using deep neural networks. Our key idea is to train a conditional generative adversarial network based on existing datasets of time-lapse videos and image sequences. We propose a multiframe joint conditional generation framework to effectively learn the correlation between the illumination change of an outdoor scene and the time of the day. We further present a multi-domain training scheme for robust training of our generative models from two datasets with different distributions and missing timestamp labels. Compared to alternative time-lapse video synthesis algorithms, our method uses the timestamp as the control variable and does not require a reference video to guide the synthesis of the final output. We conduct ablation studies to validate our algorithm and compare with state-of-the-art techniques both qualitatively and quantitatively.

show abstract

CGIntrinsics: Better Intrinsic Image Decomposition Through Physically-Based Rendering

Snavely

2018

Lecture Notes in Computer Science

Self Cite

125

184

View full text Add to dashboard Cite

Intrinsic image decomposition is a challenging, long-standing computer vision problem for which ground truth data is very difficult to acquire. We explore the use of synthetic data for training CNN-based intrinsic image decomposition models, then applying these learned models to real-world images. To that end, we present CGINTRINSICS, a new, large-scale dataset of physically-based rendered images of scenes with full ground truth decompositions. The rendering process we use is carefully designed to yield high-quality, realistic images, which we find to be crucial for this problem domain. We also propose a new end-to-end training method that learns better decompositions by leveraging CGINTRINSICS, and optionally IIW and SAW, two recent datasets of sparse annotations on realworld images. Surprisingly, we find that a decomposition network trained solely on our synthetic data outperforms the state-of-the-art on both IIW and SAW, and performance improves even further when IIW and SAW data is added during training. Our work demonstrates the suprising effectiveness of carefully-rendered synthetic data for the intrinsic images task.

show abstract

Learning Intrinsic Image Decomposition from Watching the World

Cited by 146 publications

References 38 publications

InverseRenderNet: Learning Single Image Inverse Rendering

InverseRenderNet: Learning Single Image Inverse Rendering

End-To-End Time-Lapse Video Synthesis From a Single Outdoor Image

CGIntrinsics: Better Intrinsic Image Decomposition Through Physically-Based Rendering

Contact Info

Product

Resources

About