Interactive light source position estimation for augmented reality with an RGB‐D camera

Boom, Bas; Orts-Escolano, Sergio; Ning, Xin; McDonagh, Steven; Sandilands, Peter; Fisher, Robert B.

doi:10.1002/cav.1686

Cited by 12 publications

(5 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These methods typically use image features which are known to be directly affected by illumination. Examples of such features are shadows [28], gradient of image brightness [2,3,18] and shading [10,11,14,21,26,31,32]. Real-world illumination can be also reconstructed from RGB-D images by utilizing the estimation of surface normals and albedo [33].…”

Section: Related Workmentioning

confidence: 99%

DeepLight: light source estimation for augmented reality using deep learning

Kán

Kafumann

2019

Vis Comput

View full text Add to dashboard Cite

This paper presents a novel method for illumination estimation from RGB-D images. The main focus of the proposed method is to enhance visual coherence in augmented reality applications by providing accurate and temporally coherent estimates of real illumination. For this purpose, we designed and trained a deep neural network which calculates a dominant light direction from a single RGB-D image. Additionally, we propose a novel method for real-time outlier detection to achieve temporally coherent estimates. Our method for light source estimation in augmented reality was evaluated on the set of real scenes. Our results demonstrate that the neural network can successfully estimate light sources even in scenes which were not seen by the network during training. Moreover, we compared our results with illumination estimates calculated by the state-of-the-art method for illumination estimation. Finally, we demonstrate the applicability of our method on numerous augmented reality scenes. Keywords Light source estimation • Augmented reality • Photometric registration • Deep learning (TUW). This research was funded by the Austrian research project WWTF ICT15-015. We thank Marc-Andé Gardner for providing us with the results of his algorithm through a Web service and for the kind explanations of details of the algorithm. We are also thankful to Alexander Pacha for advices about deep learning. We would like to thank NVIDIA Corporation for the donation of a Titan Xp graphics card and the Center for Geometry and Computational Design for access to a multi-GPU PC for training our neural networks. We also thank Min Kyung Lee, Iana Podkosova and Khrystyna Vasylevska for their support with controlled light experiments. Compliance with ethical standards Conflict of interest The authors declare that they have no conflict of interest. Ethical standard This research did not involve human participants.

show abstract

Section: Related Workmentioning

confidence: 99%

DeepLight: light source estimation for augmented reality using deep learning

Kán

Kafumann

2019

Vis Comput

View full text Add to dashboard Cite

show abstract

“…The auxiliary information can also be assumptions of some image features which are known to be directly affected by illumination, or simpler models like Lambertian illumination. Shadows, gradient of image brightness [35,36,115] and shading [80,81,105,145,203,251] are the typical image features used for estimating illumination direction.…”

Section: Adaptive Ui For Marmentioning

confidence: 99%

Mobile Augmented Reality: User Interfaces, Frameworks, and Intelligence

Cao,

Lam,

Lee

et al. 2021

Preprint

View full text Add to dashboard Cite

Mobile Augmented Reality (MAR) integrates computer-generated virtual objects with physical environments for mobile devices. MAR systems enable users to interact with MAR devices, such as smartphones and headworn wearables, and performs seamless transitions from the physical world to a mixed world with digital entities. These MAR systems support user experiences by using MAR devices to provide universal accessibility to digital contents. Over the past 20 years, a number of MAR systems have been developed, however, the studies and design of MAR frameworks have not yet been systematically reviewed from the perspective of user-centric design. This article presents the first effort of surveying existing MAR frameworks (count: 37) and further discusses the latest studies on MAR through a top-down approach: (1) MAR applications; (2) MAR visualisation techniques adaptive to user mobility and contexts; (3) systematic evaluation of MAR frameworks including supported platforms and corresponding features such as tracking, feature extraction plus sensing capabilities; and (4) underlying machine learning approaches supporting intelligent operations within MAR systems. Finally, we summarise the development of emerging research fields, current state-of-the-art, and discuss the important open challenges and possible theoretical and technical directions. This survey aims to benefit both researchers and MAR system developers alike. CCS Concepts: • Human-centered computing → Mixed / augmented reality; Graphical user interfaces; • Software and its engineering → Development frameworks and environments; • Computing methodologies → Machine learning approaches.

show abstract

“…Human faces can be used as light probes for coherent rendering of virtual content [29]. Depth maps can provide supplementary information to reconstruct light-source positions in a scene recorded by an RGB-D camera [30]. Convolutional Neural Networks (CNNs) enabled a related scheme for relighting using every-day objects [31].…”

Section: Scene Relightingmentioning

confidence: 99%

Enhanced Shadow Retargeting with Light-Source Estimation Using Flat Fresnel Lenses

et al. 2019

View full text Add to dashboard Cite

Shadow-retargeting maps depict the appearance of real shadows to virtual shadows given corresponding deformation of scene geometry, such that appearance is seamlessly maintained. By performing virtual shadow reconstruction from unoccluded real-shadow samples observed in the camera frame, this method efficiently recovers deformed shadow appearance. In this manuscript, we introduce a light-estimation approach that enables light-source detection using flat Fresnel lenses that allow this method to work without a set of pre-established conditions. We extend the adeptness of this approach by handling scenarios with multiple receiver surfaces and a non-grounded occluder with high accuracy. Results are presented on a range of objects, deformations, and illumination conditions in real-time Augmented Reality (AR) on a mobile device. We demonstrate the practical application of the method in generating otherwise laborious in-betweening frames for 3D printed stop-motion animation.

show abstract

Interactive light source position estimation for augmented reality with an RGB‐D camera

Cited by 12 publications

References 40 publications

DeepLight: light source estimation for augmented reality using deep learning

DeepLight: light source estimation for augmented reality using deep learning

Mobile Augmented Reality: User Interfaces, Frameworks, and Intelligence

Enhanced Shadow Retargeting with Light-Source Estimation Using Flat Fresnel Lenses

Contact Info

Product

Resources

About