SA-AE for Any-to-Any Relighting

Hu, Zhongyun; Huang, Xin; Li, Yaning; Wang, Qing

doi:10.1007/978-3-030-67070-2_32

Cited by 8 publications

(12 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Any-to-any Relighting: We modify our AMIDR-Net by changing the number of input channels and removing the skip connections corresponding to the depth maps and train it on VIDIT 2020 dataset. We compare our modified AMIDR-Net with state of the art in Table 6, where SA-AE [19] is the winner of AIM 2020 any-to-any relighting track and [9] is an encoder-decoder network proposed by another participant of the same challenge. We also compare our method with an adapted version of [53], which is originally proposed for portrait relighting.…”

Section: Comparison With State-of-the-art Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Physically Inspired Dense Fusion Networks for Relighting

Yazdani¹,

Guo²,

Monga³

2021

Preprint

View full text Add to dashboard Cite

Image relighting has emerged as a problem of significant research interest inspired by augmented reality applications. Physics-based traditional methods, as well as black box deep learning models, have been developed. The existing deep networks have exploited training to achieve a new state of the art; however, they may perform poorly when training is limited or does not represent problem phenomenology, such as the addition or removal of dense shadows. We propose a model which enriches neural networks with physical insight. More precisely, our method generates the relighted image with new illumination settings via two different strategies and subsequently fuses them using a weight map (w). In the first strategy, our model predicts the material reflectance parameters (albedo) and illumination/geometry parameters of the scene (shading) for the relit image (we refer to this strategy as intrinsic image decomposition (IID)). The second strategy is solely based on the black box approach, where the model optimizes its weights based on the ground-truth images and the loss terms in the training stage and generates the relit output directly (we refer to this strategy as direct). While our proposed method applies to both one-to-one and any-to-any relighting problems, for each case we introduce problemspecific components that enrich the model performance: 1) For one-to-one relighting we incorporate normal vectors of the surfaces in the scene to adjust gloss and shadows accordingly in the image. 2) For any-to-any relighting, we propose an additional multiscale block to the architecture to enhance feature extraction. Experimental results on the VIDIT 2020 and the VIDIT 2021 dataset (used in the NTIRE 2021 relighting challenge) reveals that our proposal can outperform many state-of-the-art methods in terms of wellknown fidelity metrics and perceptual loss.

show abstract

Section: Comparison With State-of-the-art Methodsmentioning

confidence: 99%

“…2) Direct Relighting (DR): In addition to the intrinsic decomposition of the images, we also follow the end-to-end learning method as in state of the art [19,7,38] for learning a mapping function between the two lighting settings: f (I) = I direct-relit . Where f denotes the mapping function learned by neural network model.…”

Section: Fusion Strategymentioning

confidence: 99%

Physically Inspired Dense Fusion Networks for Relighting

Yazdani¹,

Guo²,

Monga³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…First, the illuminant direction and the temperature are pre-defined [2,6], which is known as one-to-one relighting. Second, the ambient condition is based on a guided image [3], which is known as any-to-any relighting. Both of them are very similar to other low-level vision tasks like image dehazing [14] image deraining [13], image smoke removal [19], image desnowing [20], reflection removal [21], and underwater image enhancement [12].…”

Section: Deep Learning Based Image Relightingmentioning

confidence: 99%

“…On the other hand, the style transferring focuses on the texture rendering. To address this issue, recently, many deep learning-based image relighting algorithms [2,3,4,5,6] are proposed because the deep convolutional neural networks (CNNs) have achieved a lot of successful improvements in many computer vision tasks. They develop the neural networks and follow the endto-end manner to directly generate relit images without assuming any physical priors.…”

Section: Introductionmentioning

confidence: 99%

S3Net: A Single Stream Structure for Depth Guided Image Relighting

Yang¹,

Chen²,

Kuo³

2021

Preprint

View full text Add to dashboard Cite

Depth guided any-to-any image relighting aims to generate a relit image from the original image and corresponding depth maps to match the illumination setting of the given guided image and its depth map. To the best of our knowledge, this task is a new challenge that has not been addressed in the previous literature. To address this issue, we propose a deep learning-based neural Single Stream Structure network called S3Net for depth guided image relighting. This network is an encoder-decoder model. We concatenate all images and corresponding depth maps as the input and feed them into the model. The decoder part contains the attention module and the enhanced module to focus on the relighting-related regions in the guided images. Experiments performed on challenging benchmark show that the proposed model achieves the 3 rd highest SSIM in the NTIRE 2021 Depth Guided Any-to-any Relighting Challenge.

show abstract

“…As shown in Fig. 10, the team presents the novel Self-Attention AutoEncoder (SA-AE) [21] model for generating a relit image from a source image to match the illumination settings of a guide image. In order to reduce the learning difficulty, the team adopts an implicit scene representation [59] learned by the encoder to render the relit images using the decoder.…”

Section: Other Submitted Solutionmentioning

confidence: 99%

AIM 2020: Scene Relighting and Illumination Estimation Challenge

Helou¹,

Zhou²,

Süsstrunk³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

We review the AIM 2020 challenge on virtual image relighting and illumination estimation. This paper presents the novel VIDIT dataset used in the challenge and the different proposed solutions and final evaluation results over the 3 challenge tracks. The first track considered one-to-one relighting; the objective was to relight an input photo of a scene with a different color temperature and illuminant orientation (i.e., light source position). The goal of the second track was to estimate illumination settings, namely the color temperature and orientation, from a given image. Lastly, the third track dealt with any-to-any relighting, thus a generalization of the first track. The target color temperature and orientation, rather than being pre-determined, are instead given by a guide image. Participants were allowed to make use of their track 1 and 2 solutions for track 3. The tracks had 94, 52, and 56 registered participants, respectively, leading to 20 confirmed submissions in the final competition stage.

show abstract

SA-AE for Any-to-Any Relighting

Cited by 8 publications

References 24 publications

Physically Inspired Dense Fusion Networks for Relighting

Physically Inspired Dense Fusion Networks for Relighting

S3Net: A Single Stream Structure for Depth Guided Image Relighting

AIM 2020: Scene Relighting and Illumination Estimation Challenge

Contact Info

Product

Resources

About