Reconstructing Reflection Maps Using a Stacked-CNN for Mixed Reality Rendering

Chalmers, Andrew; Zhao, Junhong; Medeiros, Daniel; Rhee, Taehyun

doi:10.1109/tvcg.2020.3001917

Cited by 20 publications

(8 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, the fidelity of the relighting results generated by different methods are subjectively evaluated by a user study. Results of (Gardner et al 2017(Gardner et al , 2019Chalmers et al 2020;Zhao, Chalmers, and Rhee 2021) were provided by the authors, and the method of (Zhan et al 2021) was implemented by ourselves.…”

Section: Methodsmentioning

confidence: 99%

“…(Gardner et al 2019) uses simplified spherical Gaussian model, lacking of the details of the scene, thus produces unrealistic rendering with low roughness material. Without the illumination guidance, (Chalmers et al 2020;Zhao, Chalmers, and Rhee 2021) directly predict blurry environment maps and result in unrealistic specular reflections. Our method can predicts more accurate light directions and low-frequency lighting than (Zhan et al 2021), with the Transformer-based SG regression module and the guidance of irradiance map, which produces plausible and realistic renderings with accurate shades and shadows.…”

Section: Qualitative Evaluationmentioning

confidence: 99%

“…We compare our framework with four state-of-the-art methods that directly generate environment maps (Gardner et al 2017;Chalmers et al 2020;Zhao, Chalmers, and Rhee 2021) or estimate representative illumination functions (Gardner et al 2019). We render 300 objects with SVBRDF textures using the environment maps predicted from the test set, and show relighting errors of each method in Table 1.…”

Section: Quantitative Evaluation and User Studymentioning

confidence: 99%

“…Meanwhile, this simplified lighting parametric model can not restore the details of the scene except for main light sources , which are vital to render objects with high specular reflections, thus gets low scores in the user study. In (Gardner et al 2017;Chalmers et al 2020;Zhao, Chalmers, and Rhee 2021), direct generation of environment maps without any guidance tends to overfit to the training data and generate blurry results. In contrast, our method regresses SG coefficients for main light sources, SH coefficients for low-frequency lighting, then uses a generative network to predict high-quality environment maps under the guidance of the estimated lighting parameters.…”

Section: Quantitative Evaluation and User Studymentioning

confidence: 99%

See 3 more Smart Citations

Rendering-Aware HDR Environment Map Prediction from a Single Image

Zuo

Zhang

et al. 2022

AAAI

View full text Add to dashboard Cite

High dynamic range (HDR) illumination estimation from a single low dynamic range (LDR) image is a significant task in computer vision, graphics, and augmented reality. We present a two-stage deep learning-based method to predict an HDR environment map from a single narrow field-of-view LDR image. We first learn a hybrid parametric representation that sufficiently covers high- and low-frequency illumination components in the environment. Taking the estimated illuminations as guidance, we build a generative adversarial network to synthesize an HDR environment map that enables realistic rendering effects. We specifically consider the rendering effect by supervising the networks using rendering losses in both stages, on the predicted environment map as well as the hybrid illumination representation. Quantitative and qualitative experiments demonstrate that our approach achieves lower relighting errors for virtual object insertion and is preferred by users compared to state-of-the-art methods.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Qualitative Evaluationmentioning

confidence: 99%

Section: Quantitative Evaluation and User Studymentioning

confidence: 99%

Section: Quantitative Evaluation and User Studymentioning

confidence: 99%

See 2 more Smart Citations

Rendering-Aware HDR Environment Map Prediction from a Single Image

Zuo

Zhang

et al. 2022

AAAI

View full text Add to dashboard Cite

show abstract

“…The recent works estimate lighting by regressing representation parameters or generating illumination maps [21,37,16,53,23,6,31]. For example, Cheng et al [7] regresses the SH parameters of global lighting with a render loss.…”

Section: Related Workmentioning

confidence: 99%

Sparse Needlets for Lighting Estimation with Spherical Transport Loss

Zhan

Zhang

et al. 2021

2021 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

Accurate lighting estimation is challenging yet critical to many computer vision and computer graphics tasks such as high-dynamic-range (HDR) relighting. Existing approaches model lighting in either frequency domain or spatial domain which is insufficient to represent the complex lighting conditions in scenes and tends to produce inaccurate estimation. This paper presents NeedleLight, a new lighting estimation model that represents illumination with needlets and allows lighting estimation in both frequency domain and spatial domain jointly. An optimal thresholding function is designed to achieve sparse needlets which trims redundant lighting parameters and demonstrates superior localization properties for illumination representation. In addition, a novel spherical transport loss is designed based on optimal transport theory which guides to regress lighting representation parameters with consideration of the spatial information. Furthermore, we propose a new metric that is concise yet effective by directly evaluating the estimated illumination maps rather than rendered images. Extensive experiments show that NeedleLight achieves superior lighting estimation consistently across multiple evaluation metrics as compared with state-of-the-art methods.

show abstract

The use of CNNs in VR/AR/MR/XR: a systematic literature review

Cortes,

Bermejo,

Juiz

2024

Virtual Reality

View full text Add to dashboard Cite

This study offers a systematic literature review on the application of Convolutional Neural Networks in Virtual Reality, Augmented Reality, Mixed Reality, and Extended Reality technologies. We categorise these applications into three primary classifications: interaction, where the networks amplify user engagements with virtual and augmented settings; creation, showcasing the networks’ ability to assist in producing high-quality visual representations; and execution, emphasising the optimisation and adaptability of apps across diverse devices and situations. This research serves as a comprehensive guide for academics, researchers, and professionals in immersive technologies, offering profound insights into the cross-disciplinary realm of network applications in these realities. Additionally, we underscore the notable contributions concerning these realities and their intersection with neural networks.

show abstract

Reconstructing Reflection Maps Using a Stacked-CNN for Mixed Reality Rendering

Cited by 20 publications

References 33 publications

Rendering-Aware HDR Environment Map Prediction from a Single Image

Rendering-Aware HDR Environment Map Prediction from a Single Image

Sparse Needlets for Lighting Estimation with Spherical Transport Loss

The use of CNNs in VR/AR/MR/XR: a systematic literature review

Contact Info

Product

Resources

About