CGIntrinsics: Better Intrinsic Image Decomposition Through Physically-Based Rendering

Abstract

Intrinsic image decomposition is a challenging, long-standing computer vision problem for which ground truth data is very difficult to acquire. We explore the use of synthetic data for training CNN-based intrinsic image decomposition models, then applying these learned models to real-world images. To that end, we present CGINTRINSICS, a new, large-scale dataset of physically-based rendered images of scenes with full ground truth decompositions. The rendering process we use is carefully designed to yield high-quality, realistic images, which we find to be crucial for this problem domain. We also propose a new end-to-end training method that learns better decompositions by leveraging CGINTRINSICS, and optionally IIW and SAW, two recent datasets of sparse annotations on realworld images. Surprisingly, we find that a decomposition network trained solely on our synthetic data outperforms the state-of-the-art on both IIW and SAW, and performance improves even further when IIW and SAW data is added during training. Our work demonstrates the suprising effectiveness of carefully-rendered synthetic data for the intrinsic images task.

Photorealistic Image Synthesis for Object Instance Detection

“…Physically based rendering (PBR) techniques can synthesize images with a high degree of visual realism which promises to reduce the domain gap. Li and Snavely [8] used PBR images to train models for intrinsic image decomposition and Zhang et al [9] for semantic segmentation, normal estimation and boundary detection. However, they focus on scene understanding not object understanding tasks.…”

Deep CG2Real: Synthetic-to-Real Translation via Image Disentanglement

“…Our intrinsic image model assumes Lambertian shading. While this is an approximation to real world reflectance, it is sufficient for many regions of indoor images, and is widely used [2,3,23]. To recover high dynamic shading values, we predict the shading in the logarithmic space.…”

“…However, translating it back to the real domain via the forward p → r translation requires an albedo-shading separation that we do not have for these images-we only have them for our original synthetic images. We tackle this by training an intrinsic decomposition network [23,3,4] to predict the albedo and shading layers for PBR images. Let H be the intrinsic decomposition network.…”

Block Annotation: Better Image Annotation With Sub-Image Decomposition

“…While human annotations may contain mis- takes, synthetic datasets are generated with known ground truth labels with which annotation error can be computed. The CGIntrinsics dataset [35] contains physically-based renderings of indoor scenes from the SUNCG dataset [54,63]. We use the more realistic CGIntrinsics renderings and the known semantic labels from SUNCG.…”