NTIRE 2021 Depth Guided Image Relighting Challenge

Helou, Majed El; Zhou, Ruofan; Süsstrunk, Sabine; Timofte, Radu; Suin, Maitreya; Rajagopalan, A. N.; Wang, Yuanzhi; Lü, Tao; Zhang, Yanduo; Wu, Yuntao; Yang, Hao-Hsiang; Chen, Wei‐Ting; Kuo, Sy‐Yen; Luo, Hao-Lun; Zhang, Zhiguang; Luo, Zhipeng; He, Jianye; Zhu, Zuo-Liang; Li, Zhen; Qiu, Jiaxiong; Kuang, Zengsheng; Lu, Chin-Pi; Cheng, Ming–Ming; Shao, Xi; Li, Chenghua; Ding, Bosong; Qian, Wanli; Li, Fangya; Li, Fu; Deng, Ruifeng; Lin, Tianwei; Liu, Songhua; Li, Xin; He, Dongliang; Yazdani, Amirsaeed; Guo, Tiantong; Monga, Vishal; Elie, Nsampi, Ntumba; Hu, Zhongyun; Wang, Qing; Nathan, Sabari; Kansal, Priya; Zhao, Tongtong; Zhao, Shanshan

doi:10.1109/cvprw53098.2021.00069

Cited by 23 publications

(32 citation statements)

References 57 publications

(69 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…General-purpose relighting. Last year the VIDIT dataset [8] was created and used for relighting challenges [9,10]. It features indoor and outdoor scenes lit from 8 directions and captured under 5 temperature settings.…”

Section: Literature Reviewmentioning

confidence: 99%

SILT: Self-supervised Lighting Transfer Using Implicit Image Decomposition

Nikolina¹,

Mustafa²,

Phillipson³

et al. 2021

Preprint

View full text Add to dashboard Cite

We present SILT, a Self-supervised Implicit Lighting Transfer method. Unlike previous research on scene relighting, we do not seek to apply arbitrary new lighting configurations to a given scene. Instead, we wish to transfer the lighting style from a database of other scenes, to provide a uniform lighting style regardless of the input. The solution operates as a two-branch network that first aims to map input images of any arbitrary lighting style to a unified domain, with extra guidance achieved through implicit image decomposition. We then remap this unified input domain using a discriminator that is presented with the generated outputs and the style reference, i.e. images of the desired illumination conditions. Our method is shown to outperform supervised relighting solutions across two different datasets without requiring lighting supervision. The code and pre-trained models can be found here. IntroductionWe propose the problem of lighting transfer where an input image under arbitrary illumination conditions is adapted to match the lighting of a style reference database. Previous approaches to a similar problem -arbitrary image relighting -either make simplifying assumptions about the scene (e.g. a single dominant object or a single light source) or require costly supervision where identical scenes must be captured under a large number of known lighting conditions. In contrast, our lighting transfer approach, SILT, is entirely self-supervised. SILT requires only a training dataset with multiple examples of the same scene and a style reference database. It is not necessary to know the ground truth lighting conditions for the training data, nor is it necessary for the target illumination to be present within the training dataset. This distinction between image relighting and lighting transfer is demonstrated in Fig. 1.The proposed SILT method consists of a two-branch network. During training, the generator sees a number of input images of the same scene under different unknown illuminations. The model attempts to enforce similarity between the illumination conditions of the outputs

show abstract

Section: Literature Reviewmentioning

confidence: 99%

SILT: Self-supervised Lighting Transfer Using Implicit Image Decomposition

Nikolina¹,

Mustafa²,

Phillipson³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Image relighting is now an established task. For scenes, there are workshop tracks (eg [21], [22]), challenges [23], [24] and datasets [25], [26]. Existing work learns image mappings (pure image mappings as in [27]; depth guided, as in [28]; using wavelets, as in [29]; shadow priors, as in [30]).…”

Section: Related Workmentioning

confidence: 99%

SIRfyN: Single Image Relighting from your Neighbors

Forsyth¹,

Bhattad²,

Asthana³

et al. 2021

Preprint

View full text Add to dashboard Cite

We show how to relight a scene, depicted in a single image, such that (a) the overall shading has changed and (b) the resulting image looks like a natural image of that scene. Applications for such a procedure include generating training data and building authoring environments. Naive methods for doing this fail. One reason is that shading and albedo are quite strongly related; for example, sharp boundaries in shading tend to appear at depth discontinuities, which usually apparent in albedo. The same scene can be lit in different ways, and established theory shows the different lightings form a cone (the illumination cone). Novel theory shows that one can use similar scenes to estimate the different lightings that apply to a given scene, with bounded expected error. Our method exploits this theory to estimate a representation of the available lighting fields in the form of imputed generators of the illumination cone. Our procedure does not require expensive "inverse graphics" datasets, and sees no ground truth data of any kind.Qualitative evaluation suggests the method can erase and restore soft indoor shadows, and can "steer" light around a scene. We offer a summary quantitative evaluation of the method with a novel application of the FID. An extension of the FID allows per-generated-image evaluation. Furthermore, we offer qualitative evaluation with a user study, and show that our method produces images that can successfully be used for data augmentation.

show abstract

“…This challenge is one of the NTIRE 2021 associated challenges: nonhomogeneous dehazing [4], defocus deblurring using dual-pixel [2], depth guided image relighting [16], image deblurring [39], multi-modal aerial view imagery classification [33], learning the super-resolution space [35], quality enhancement of compressed video (this report), video super-resolution [46], perceptual image quality assessment [18], burst super-resolution [5], and high dynamic range imaging [41].…”

Section: Ntire 2021 Challengementioning

confidence: 99%

NTIRE 2021 Challenge on Quality Enhancement of Compressed Video: Methods and Results

Yang¹,

Timofte²,

Liu³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

This paper reviews the first NTIRE challenge on quality enhancement of compressed video, with a focus on the proposed methods and results. In this challenge, the new Large-scale Diverse Video (LDV) dataset is employed. The challenge has three tracks. Tracks 1 and 2 aim at enhancing the videos compressed by HEVC at a fixed QP, while Track 3 is designed for enhancing the videos compressed by x265 at a fixed bit-rate. Besides, the quality enhancement of Tracks 1 and 3 targets at improving the fidelity (PSNR), and Track 2 targets at enhancing the perceptual quality. The three tracks totally attract 482 registrations. In the test phase, 12 teams, 8 teams and 11 teams submitted the final results of Tracks 1, 2 and 3, respectively. The proposed methods and solutions gauge the state-of-the-art of video quality enhancement. The homepage of the challenge: https: //github.com/RenYang-home/NTIRE21_VEnh

show abstract

NTIRE 2021 Depth Guided Image Relighting Challenge

Cited by 23 publications

References 57 publications

SILT: Self-supervised Lighting Transfer Using Implicit Image Decomposition

SILT: Self-supervised Lighting Transfer Using Implicit Image Decomposition

SIRfyN: Single Image Relighting from your Neighbors

NTIRE 2021 Challenge on Quality Enhancement of Compressed Video: Methods and Results

Contact Info

Product

Resources

About