DeepLight: light source estimation for augmented reality using deep learning

Abstract

This paper presents a novel method for illumination estimation from RGB-D images. The main focus of the proposed method is to enhance visual coherence in augmented reality applications by providing accurate and temporally coherent estimates of real illumination. For this purpose, we designed and trained a deep neural network which calculates a dominant light direction from a single RGB-D image. Additionally, we propose a novel method for real-time outlier detection to achieve temporally coherent estimates. Our method for light source estimation in augmented reality was evaluated on the set of real scenes. Our results demonstrate that the neural network can successfully estimate light sources even in scenes which were not seen by the network during training. Moreover, we compared our results with illumination estimates calculated by the state-of-the-art method for illumination estimation. Finally, we demonstrate the applicability of our method on numerous augmented reality scenes. Keywords Light source estimation • Augmented reality • Photometric registration • Deep learning (TUW). This research was funded by the Austrian research project WWTF ICT15-015. We thank Marc-Andé Gardner for providing us with the results of his algorithm through a Web service and for the kind explanations of details of the algorithm. We are also thankful to Alexander Pacha for advices about deep learning. We would like to thank NVIDIA Corporation for the donation of a Titan Xp graphics card and the Center for Geometry and Computational Design for access to a multi-GPU PC for training our neural networks. We also thank Min Kyung Lee, Iana Podkosova and Khrystyna Vasylevska for their support with controlled light experiments. Compliance with ethical standards Conflict of interest The authors declare that they have no conflict of interest. Ethical standard This research did not involve human participants.

Deep Light Direction Reconstruction from single RGB images

“…They integrated their network into an AR application and achieved a mean angular error of approximately 28 • and an inference time of approximately 380 ms while running on a central processing unit. The rendering equation (Section 1) suggests all required information to be present in an RGB image to reconstruct the illumination origin in a scene, however, given the unsuccessful experiment of Kán and Kaufmann (2019) using only RGB information, we begin with a simple scenario to investigate if RGB images provide enough information to reconstruct the dominant light direction. Considering the different dataset sizes and training strategies of Kán and Kaufmann (2019) and Marques et al (2018), a DNN with model weights pre-trained on a large dataset as initial values for training and a large number of training examples appears to be a reasonable basis for a regression approach that aims to reconstruct the dominant light direction from RGB images.…”

Deep Light Direction Reconstruction from single RGB images

“…With 100 possible point light sources available in the virtual scene, their network achieved a top 1 accuracy of approximately 82% in classifying the correct point light source. Kán and Kaufmann (2019) proposed a regression model to reconstruct continuous azimuth and elevation angle values indicating the dominant light direction in real scenes using RGB-D input images for their ResNet. They also conducted experiments using the RGB com-ponents of their RGB-D dataset; however, their network failed when it relied solely on these three colour channels.…”

“…In our proposed approach, we aim to reconstruct the azimuth and elevation angles (φ , θ ) of the dominant light direction from RGB input images. This is similar to the approach of Kán and Kaufmann (2019); however, our approach focuses on light direction reconstruction using only RGB information and explicitly omits the depth information. We assume that the network is capable of reconstructing the light direction without additional depth information.…”

“…Our approach exceeds existing learning-based approaches for inferring light direction from images (Kán and Kaufmann, 2019) in that it enables reconstruction from RGB images. This is achieved by increasing the amount of training data and using pre-trained model weights, which has been successfully applied to a similar problem (Marques et al, 2018), and inferring the light direction in a stereographic coordinate representation.…”

Implementasi Augmented Reality Furniture Dengan User-Defined Target Berbasis Android

“…Pemanfaatannya dapat dirasakan dalam bidang pendidikan tinggi (Nguyen, et al, 2018), untuk pembelajaran keahlian bedah syaraf (Si, et al, 2019), perancangan brosur interaktif (Rumajar, et al, 2015), sarana promosi produk tertentu (Tijono, et al, 2015) dan banyak aplikasi lainnya. Dalam riset lain, penggabungan AR dan deep learning digunakan untuk mengestimasi pencahayaan benda (object illumination) pada lingkungan gambar nyata (Kán & Kafumann, 2019). Dalam penelitian tersebut, arah sumber cahaya disimulasi dengan deep learning sehingga menghasilkan arah bayangan yang tepat.…”