Region-aware Adaptive Instance Normalization for Image Harmonization

Ling, Jun; Han, Xue; Song, Li; Xie, Rong; Gu, Xiao

doi:10.1109/cvpr46437.2021.00924

Cited by 110 publications

(118 citation statements)

References 25 publications

Supporting

Mentioning

118

Contrasting

Order By: Relevance

“…The detailed results on different foreground ratio ranges can be found in the Supplementary. Our CDTNet-256 outperforms Ling et al 2021) by a large margin and also beats iS 2 AM. Even our simplified variant CDTNet-256(sim) outperforms most methods, which demonstrates the expressiveness of our proposed deep RGB-to-RGB transformation.…”

Section: Quantitative Comparison With Existing Methodsmentioning

confidence: 78%

“…Since there are no existing high-resolution image harmonization methods available for comparison, we transplant low-resolution image harmonization methods Sofiiuk, Popenova, and Konushin 2021;Ling et al 2021) and high-resolution image-to-image translation methods (Wang et al 2018a; to our task with essential modification of their official implementation. The low-resolution image harmonization models Sofiiuk, Popenova, and Konushin 2021;Ling et al 2021) can be trained on high-resolution images despite the huge memory consumption. Thus, we train these models on high-resolution images with sufficient GPU memory.…”

Section: Baseline Transplantationmentioning

confidence: 99%

“…To address this issue, image harmonization adjusts the foreground appearance to make it compatible with the background. Deep image harmonization methods (Tsai et al 2017;Cong et al , 2021Sofiiuk, Popenova, and Konushin 2021;Ling et al 2021) have achieved remarkable progress by learning the dense pixel-to-pixel transformation between composite images and ground-truth harmonized images. However, they only performed low-resolution (e.g., 256 × 256) image harmonization, and a naïve upsampling can merely lead to an unacceptably blurry output (see Figure 1a).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

High-Resolution Image Harmonization via Collaborative Dual Transformations

Cong¹,

Xinhao²,

Niu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Given a composite image, image harmonization aims to adjust the foreground to make it compatible with the background. High-resolution image harmonization is in high demand, but still remains unexplored. Conventional image harmonization methods learn global RGB-to-RGB transformation which could effortlessly scale to high resolution, but ignore diverse local context. Recent deep learning methods learn the dense pixel-to-pixel transformation which could generate harmonious outputs, but are highly constrained in low resolution. In this work, we propose a high-resolution image harmonization network with Collaborative Dual Transformation (CDTNet) to combine pixel-to-pixel transformation and RGB-to-RGB transformation coherently in an endto-end framework. Our CDTNet consists of a low-resolution generator for pixel-to-pixel transformation, a color mapping module for RGB-to-RGB transformation, and a refinement module to take advantage of both. Extensive experiments on high-resolution image harmonization dataset demonstrate that our CDTNet strikes a good balance between efficiency and effectiveness.

show abstract

Section: Quantitative Comparison With Existing Methodsmentioning

confidence: 78%

Section: Baseline Transplantationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

High-Resolution Image Harmonization via Collaborative Dual Transformations

Cong¹,

Xinhao²,

Niu³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Existing deep image harmonization methods [139,25,31,56,88,96] could be divided into supervised methods and unsupervised methods depending on whether using paired training data.…”

Section: Deep Learning Methodsmentioning

confidence: 99%

“…Guo et al [54] realizes that inharmony derives from intrinsic reflectance and illumination difference of foreground and background, and propose an autoencoder to disentangle composite image into reflectance and illumination for separate harmonization, where reflectance is harmonized through material-consistency penalty and illumination is harmonized by learning and transferring light from background to foreground. In [88], they reframe image harmonization as a background-to-foreground style transfer problem and propose region-aware adaptive instance normalization module to explicitly formulate the visual style from the background and adaptively apply them to the foreground.…”

Section: Deep Learning Methodsmentioning

confidence: 99%

Making Images Real Again: A Comprehensive Survey on Deep Image Composition

Niu¹,

Cong²,

Liu³

et al. 2021

Preprint

View full text Add to dashboard Cite

As a common image editing operation, image composition aims to cut the foreground from one image and paste it on another image, resulting in a composite image. However, there are many issues that could make the composite images unrealistic. These issues can be summarized as the inconsistency between foreground and background, which include appearance inconsistency (e.g., incompatible color and illumination) and geometry inconsistency (e.g., unreasonable size and location). Previous works on image composition target at one or more issues. Since each individual issue is a complicated problem, there are some research directions (e.g., image harmonization, object placement) which focus on only one issue. By putting all the efforts together, we can acquire realistic composite images. Sometimes, we expect the composite images to be not only realistic but also aesthetic, in which case aesthetic evaluation needs to be considered. In this survey, we summarize the datasets and methods for the above research directions. We also discuss the limitations and potential directions to facilitate the future research for image composition. Finally, as a double-edged sword, image composition may also have negative effect on our lives (e.g., fake news) and thus it is imperative to develop algorithms to fight against composite images. Datasets and codes for image composition are summarized at https://github.com/bcmi/Awesome-Image-Composition.

show abstract

Harmonizer: Learning to Perform White-Box Image and Video Harmonization

Sun

Zhu

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Region-aware Adaptive Instance Normalization for Image Harmonization

Cited by 110 publications

References 25 publications

High-Resolution Image Harmonization via Collaborative Dual Transformations

High-Resolution Image Harmonization via Collaborative Dual Transformations

Making Images Real Again: A Comprehensive Survey on Deep Image Composition

Harmonizer: Learning to Perform White-Box Image and Video Harmonization

Contact Info

Product

Resources

About