Spatiotemporal Outdoor Lighting Aggregation on Image Sequences

Lee, Haebom; Herzog, Robert; Rexilius, Jan; Rother, Carsten

doi:10.1007/978-3-030-92659-5_22

Cited by 2 publications

(29 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We evaluate the angular errors of the spatio-temporally aggregated sun direction estimates on the SUN360 test sequences. At first, we present single image lighting estimation results from [9], [10], [29], and [14]. On top of that, we compare our method with the spatio-temporal aggregation pipeline proposed in [14].…”

Section: Resultsmentioning

confidence: 99%

“…In order to extract robust estimates from noisy observations, the aggregation process described in [14] relies heavily on statistical filtering utilizing an outlier removal combined with the meanshift algorithm. However, this approach requires manual hyper-parameter tuning with handcrafted selection criteria.…”

Section: Attention Based Aggregationmentioning

confidence: 99%

“…This includes self attention using multi-head attention layers [33] and preprocessing images with a pretrained convolutional neural network. Given a temporal sequence of k images, we first select n spatially random crops for each frame as done in our previous work [14]. On each crop, we apply a ResNet18 [32] encoder to extract feature embeddings.…”

Section: Attention Based Aggregationmentioning

confidence: 99%

“…Several previous methods utilized it in its original panorama form or as subimages by generating synthetic perspective images [9]. We followed the latter approach in our preliminary work [14] where we examined the performance improvement arising from the spatial aggregation. In particular, we simulate a camera motion without translation by generating a set of synthetic perspective images with a fixed field of view and randomized camera yaw and pitch angles.…”

Section: Datasetsmentioning

confidence: 99%

“…A preliminary version of this work has been published in [14]. In this paper, we extend that work by using an end-2-end filtering approach that supersedes the statistical post-processing in [14] by using a Transformer architecture [15,16,17] which accounts for individual orientations and field-of-views of the input frames. In our experiments in Section 4, we replace parts of our estimation pipeline and adapt the architecture of [15] for lighting source regression.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Spatio-Temporal Outdoor Lighting Aggregation on Image Sequences using Transformer Networks

Lee¹,

Homeyer²,

Herzog³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

In this work, we focus on outdoor lighting estimation by aggregating individual noisy estimates from images, exploiting the rich image information from wide-angle cameras and/or temporal image sequences. Photographs inherently encode information about the scene's lighting in the form of shading and shadows. Recovering the lighting is an inverse rendering problem and as that ill-posed. Recent work based on deep neural networks has shown promising results for single image lighting estimation, but suffers from robustness. We tackle this problem by combining lighting estimates from several image views sampled in the angular and temporal domain of an image sequence. For this task, we introduce a transformer architecture that is trained in an end-2-end fashion without any statistical post-processing as required by previous work. Thereby, we propose a positional encoding that takes into account the camera calibration and ego-motion estimation to globally register the individual estimates when computing attention between visual words. We show that our method leads to improved lighting estimation while requiring less hyper-parameters compared to the state-of-the-art.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Attention Based Aggregationmentioning

confidence: 99%

Section: Attention Based Aggregationmentioning

confidence: 99%

Section: Datasetsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Spatio-Temporal Outdoor Lighting Aggregation on Image Sequences using Transformer Networks

Lee¹,

Homeyer²,

Herzog³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Spatio-Temporal Outdoor Lighting Aggregation on Image Sequences Using Transformer Networks

et al. 2022

Self Cite

View full text Add to dashboard Cite

In this work, we focus on outdoor lighting estimation by aggregating individual noisy estimates from images, exploiting the rich image information from wide-angle cameras and/or temporal image sequences. Photographs inherently encode information about the lighting of the scene in the form of shading and shadows. Recovering the lighting is an inverse rendering problem and as that ill-posed. Recent research based on deep neural networks has shown promising results for estimating light from a single image, but with shortcomings in robustness. We tackle this problem by combining lighting estimates from several image views sampled in the angular and temporal domains of an image sequence. For this task, we introduce a transformer architecture that is trained in an end-2-end fashion without any statistical post-processing as required by previous work. Thereby, we propose a positional encoding that takes into account camera alignment and ego-motion estimation to globally register the individual estimates when computing attention between visual words. We show that our method leads to improved lighting estimation while requiring fewer hyperparameters compared to the state of the art.

show abstract

Spatiotemporal Outdoor Lighting Aggregation on Image Sequences

Cited by 2 publications

References 29 publications

Spatio-Temporal Outdoor Lighting Aggregation on Image Sequences using Transformer Networks

Spatio-Temporal Outdoor Lighting Aggregation on Image Sequences using Transformer Networks

Spatio-Temporal Outdoor Lighting Aggregation on Image Sequences Using Transformer Networks

Contact Info

Product

Resources

About