Learning to Predict Indoor Illumination from a Single Image

Gardner, Marc-André; Sunkavalli, Kalyan; Yumer, Ersin; Shen, Xiaohui; Gambaretto, Emiliano; Gagné, Christian; Lalonde, Jean‐François

doi:10.48550/arxiv.1704.00090

Cited by 17 publications

(46 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Li et al [20] represent illumination maps with multiple spherical Gaussian functions and regresses the corresponding Gaussian parameters for lighting estimation. Gardner et al [14] generate illumination maps directly with a two-steps training strategy. Legendre et al [19] regress HDR lighting from LDR images by comparing the ground-truth sphere image to the rendered one with the predicted illumination.…”

Section: Related Workmentioning

confidence: 99%

“…We evaluate NeedleLight by using the Laval Indoor HDR Dataset [14] that consists of 2,100 HDR panoramas taken in a variety of indoor environments. Similar to [14], we extract eight limited field of view crops from each panorama which produces 19,556 images as used in our experiments. The image warping operation as described in [14] is applied to the panoramas.…”

Section: Dataset and Implementationmentioning

confidence: 99%

“…Similar to [14], we extract eight limited field of view crops from each panorama which produces 19,556 images as used in our experiments. The image warping operation as described in [14] is applied to the panoramas. We apply the proposed sparse needlets with j max = 3 to extract needlet coefficients as the ground truth for training.…”

Section: Dataset and Implementationmentioning

confidence: 99%

“…We compare NeedleLight with a number of state-of-theart lighting estimation methods including Garon et al [15] that estimates lighting in frequency domain and Gardner et al [14,13] and Li et al [20] that estimate lighting in spatial domain. To perform quantitative evaluations, we create three spheres with gray diffuse, matte silver and mirror silver materials for evaluation as shown in Fig.…”

Section: Quantitative Evaluationmentioning

confidence: 99%

“…Lighting estimation has been tackled by regressing the parameters of various lighting representations in either frequency domain [7,15]) or spatial domain [14,13,34,35]. However, lighting estimation in frequency domain normally represents illumination with Spherical Harmonics (SH) which lack spatial localization capabilities.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Sparse Needlets for Lighting Estimation with Spherical Transport Loss

Zhan

Zhang

et al. 2021

2021 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

Accurate lighting estimation is challenging yet critical to many computer vision and computer graphics tasks such as high-dynamic-range (HDR) relighting. Existing approaches model lighting in either frequency domain or spatial domain which is insufficient to represent the complex lighting conditions in scenes and tends to produce inaccurate estimation. This paper presents NeedleLight, a new lighting estimation model that represents illumination with needlets and allows lighting estimation in both frequency domain and spatial domain jointly. An optimal thresholding function is designed to achieve sparse needlets which trims redundant lighting parameters and demonstrates superior localization properties for illumination representation. In addition, a novel spherical transport loss is designed based on optimal transport theory which guides to regress lighting representation parameters with consideration of the spatial information. Furthermore, we propose a new metric that is concise yet effective by directly evaluating the estimated illumination maps rather than rendered images. Extensive experiments show that NeedleLight achieves superior lighting estimation consistently across multiple evaluation metrics as compared with state-of-the-art methods.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Dataset and Implementationmentioning

confidence: 99%

Section: Dataset and Implementationmentioning

confidence: 99%

Section: Quantitative Evaluationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Sparse Needlets for Lighting Estimation with Spherical Transport Loss

Zhan

Zhang

et al. 2021

2021 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

show abstract

Online Illumination Learning for Interactive Global Illumination in Augmented Reality

et al. 2022

View full text Add to dashboard Cite

Augmenting virtual objects into a real scene requires estimating the scene illumination so that the augmented objects can become visually coherent with real objects. We propose an online technique that learns the illumination from image sequences captured by a hand-held device. We approximate the illumination with multiple linear models, and the coefficients and bandwidth parameters of the models are updated progressively in a data-driven way. Our online learning enables us to seamlessly integrate virtual objects into a real scene by rendering the objects with the estimated lights. We demonstrate that our framework can provide a high-quality global illumination result in augmented reality at interactive rates.

show abstract

Scene-Aware Audio Rendering via Deep Acoustic Analysis

Tang

Bryan

et al. 2020

IEEE Trans. Visual. Comput. Graphics

View full text Add to dashboard Cite

Fig. 1: Given a natural sound in a real-world room that is recorded using a cellphone microphone (left), we estimate the acoustic material properties and the frequency equalization of the room using a novel deep learning approach (middle). We use the estimated acoustic material properties for generating plausible sound effects in the virtual model of the room (right). Our approach is general and robust, and works well with commodity devices.Abstract-We present a new method to capture the acoustic characteristics of real-world rooms using commodity devices, and use the captured characteristics to generate similar sounding sources with virtual models. Given the captured audio and an approximate geometric model of a real-world room, we present a novel learning-based method to estimate its acoustic material properties. Our approach is based on deep neural networks that estimate the reverberation time and equalization of the room from recorded audio. These estimates are used to compute material properties related to room reverberation using a novel material optimization objective. We use the estimated acoustic material characteristics for audio rendering using interactive geometric sound propagation and highlight the performance on many real-world scenarios. We also perform a user study to evaluate the perceptual similarity between the recorded sounds and our rendered audio.

show abstract

Learning to Predict Indoor Illumination from a Single Image

Cited by 17 publications

References 0 publications

Sparse Needlets for Lighting Estimation with Spherical Transport Loss

Sparse Needlets for Lighting Estimation with Spherical Transport Loss

Online Illumination Learning for Interactive Global Illumination in Augmented Reality

Scene-Aware Audio Rendering via Deep Acoustic Analysis

Contact Info

Product

Resources

About