GLEAM -- An Illumination Estimation Framework for Real-time Photorealistic Augmented Reality on Mobile Devices (demo)

Prakash, Siddhant; Bahremand, Alireza; Nguyen, Linda D.; LiKamWa, Robert

doi:10.1145/3307334.3328570

Cited by 13 publications

(25 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our edge service is on a desktop running Ubuntu 20.04 with a 16 core AMD Ryzen Threadripper 2950X CPU, 128GB memory, and a Nvidia RTX 2080Ti GPU. We quantified Xihe's performance in terms of end-to-end lighting estimation time, accuracy, and visual effects and compared it to the commercial baseline ARKit [13], an academic framework GLEAM [21], and a 3D vision estimation pipeline [33] where appropriate. Xihe can deliver spatially-variant lighting estimation as fast as 20.67ms and achieves visually-coherent rendering effects.…”

Section: Discussionmentioning

confidence: 99%

“…indoor RGB-D dataset on several mobile devices and network conditions and show that Xihe can achieve better visual effects than two existing approaches, i.e., GLEAM [21] and ARKit [13]. Further, we also present a detailed performance breakdown of Xihe under different configurations and use cases, reporting mobile, network, and edge time.…”

Section: Introductionmentioning

confidence: 90%

“…We implement the framework Xihe on top of Unity3D and AR Foundation as well as a proof-of-concept reference AR application that utilizes Xihe's APIs. Figure 1 compares the rendered AR scenes using Xihe and prior work [21]. To support the key goal of spatially-variant lighting estimation, we design an end-to-end pipeline for 3D data processing, understanding, and management.…”

Section: Introductionmentioning

confidence: 99%

“…Spatially-variant lighting information can be traditionally extracted using physical probes [6,21], and more recently estimated with deep neural networks [7,8,28,33]. For example, Debevec et al demonstrated that spatially-variant lighting can be effectively estimated by using reflective sphere light probes to extrapolate camera views.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Xihe

Zhao

Guo

2021

Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services

View full text Add to dashboard Cite

Omnidirectional lighting provides the foundation for achieving spatially-variant photorealistic 3D rendering, a desirable property for mobile augmented reality applications. However, in practice, estimating omnidirectional lighting can be challenging due to limitations such as partial panoramas of the rendering positions, and the inherent environment lighting and mobile user dynamics. A new opportunity arises recently with the advancements in mobile 3D vision, including built-in high-accuracy depth sensors and deep learning-powered algorithms, which provide the means to better sense and understand the physical surroundings. Centering the key idea of 3D vision, in this work, we design an edge-assisted framework called Xihe to provide mobile AR applications the ability to obtain accurate omnidirectional lighting estimation in real time.Specifically, we develop a novel sampling technique that efficiently compresses the raw point cloud input generated at the mobile device. This technique is derived based on our empirical analysis of a recent 3D indoor dataset and plays a key role in our 3D vision-based lighting estimator pipeline design. To achieve the realtime goal, we develop a tailored GPU pipeline for on-device point cloud processing and use an encoding technique that reduces network transmitted bytes. Finally, we present an adaptive triggering strategy that allows Xihe to skip unnecessary lighting estimations and a practical way to provide temporal coherent rendering integration with the mobile AR ecosystem. We evaluate both the lighting estimation accuracy and time of Xihe using a reference mobile application developed with Xihe's APIs. Our results show that Xihe takes as fast as 20.67ms per lighting estimation and achieves 9.4% better estimation accuracy than a state-of-the-art neural network. CCS CONCEPTS• Computing methodologies → Mixed / augmented reality; • Human-centered computing → Ubiquitous and mobile computing systems and tools; • Computer systems organization → Distributed architectures.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 90%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Xihe

Zhao

Guo

2021

Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services

View full text Add to dashboard Cite

show abstract

“…OverLay [17] labels real-world objects with the help of odometry sensors, whereas we additionally incorporate continuous camera frames, and can display virtual objects with 3D location/orientation in the world. MARVEL [6] focuses on the energy-efficiency of mobile AR, and GLEAM [24] discusses lighting rendering for virtual objects, which are orthogonal directions to this work.…”

Section: Related Workmentioning

confidence: 99%

ShareAR

Ran

Slocum

Gorlatova

et al. 2019

Proceedings of the 18th ACM Workshop on Hot Topics in Networks

View full text Add to dashboard Cite

Augmented reality is an emerging application on mobile devices. However, there is a lack of understanding of the communication requirements and challenges of multiuser AR scenarios. In this position paper, we propose several important research issues that need to be addressed for low-latency, accurate shared AR experiences: (a) Systems tradeoffs of AR communication architectures used today in mobile AR platforms; (b) Understanding AR communication patterns and adapting the AR application layer to dynamically changing network conditions; and (c) Tools and methodologies to evaluate AR quality of experience in real time on mobile devices. We present preliminary measurements of off-the-shelf mobile AR platforms as well as results from our AR system, ShareAR, illustrating performance tradeoffs and indicating promising new research directions.

show abstract

PointAR: Efficient Lighting Estimation for Mobile Augmented Reality

Zhao

Guo

2020

Computer Vision – ECCV 2020

View full text Add to dashboard Cite

GLEAM -- An Illumination Estimation Framework for Real-time Photorealistic Augmented Reality on Mobile Devices (demo)

Cited by 13 publications

References 0 publications

Xihe

Xihe

ShareAR

PointAR: Efficient Lighting Estimation for Mobile Augmented Reality

Contact Info

Product

Resources

About