FRIH: Fine-grained Region-aware Image Harmonization

Abstract

Image harmonization aims to generate a more realistic appearance of foreground and background for a composite image. Existing methods perform the same harmonization process for the whole foreground. However, the implanted foreground always contains different appearance patterns. All the existing solutions ignore the difference of each color block and losing some specific details. Therefore, we propose a novel global-local two stages framework for Fine-grained Region-aware Image Harmonization (FRIH), which is trained end-to-end. In the first stage, the whole input foreground mask is used to make a global coarse-grained harmonization. In the second stage, we adaptively cluster the input foreground mask into several submasks by the corresponding pixel RGB values in the composite image. Each submask and the coarsely adjusted image are concatenated respectively and fed into a lightweight cascaded module, adjusting the global harmonization performance according to the region-aware local feature. Moreover, we further designed a fusion prediction module by fusing features from all the cascaded decoder layers together to generate the final result, which could utilize the different degrees of harmonization results comprehensively. Without bells and whistles, our FRIH algorithm achieves the best performance on iHarmony4 dataset (PSNR is 38.19 dB) with a lightweight model. The parameters for our model are only 11.98 M, far below the existing methods.

Semantic‐aware visual consistency network for fused image harmonisation

“…To illustrate the harmonisation performance, we compared our SVCNet with traditional methods [7][8][9][10] and some deeplearning-based methods [11][12][13][14][15][16][17][18][19][20][21][22][23]. Table 1 and Table 2 show the results for each subdataset and foreground ratio range, respectively.…”

“…Guo et al [16,17] proposed an intrinsic imageharmonisation framework that separated an image into reflectance and illumination for further processing. FRIH [18] proposed a coarse-to-fine region-aware framework that leverages several clustered submasks to refine harmonisation performance. Ren et al [19] aggregated the estimated local light into global light to guide the foreground adjustment.…”

“…To address this issue, semantic information was introduced into the harmonisation network. However, some recent studies [13][14][15][16][17][18][19][20][21][22][23] did not consider the use of semantic features. Tsai et al [11] leveraged an extra branch to capture the contextual information of images; the semantic features passing through the harmonisation network were weakened by the common normalisation layers.…”

Painterly Image Harmonization by Learning from Painterly Objects

“…Image harmonization aims to harmonize a composite image by adjusting foreground illumination to match background illumination. In recent years, abundant deep image harmonization methods (Tsai et al 2017;Jiang et al 2021;Xing et al 2022;Peng et al 2022;Zhu et al 2022;Valanarasu et al 2023;Liu et al 2023) have been developed. For example, (Cun and Pun 2020;Hao, Iizuka, and Fukui 2020;Sofiiuk, Popenova, and Konushin 2021) proposed diverse attention modules to treat the foreground and background separately, or establish the relation between foreground and background.…”

Progressive Painterly Image Harmonization from Low-Level Styles to High-Level Styles

“…Image harmonization aims to harmonize a composite image by adjusting foreground illumination to match background illumination. In recent years, abundant deep image harmonization methods (Tsai et al 2017;Jiang et al 2021;Xing et al 2022;Peng et al 2022;Zhu et al 2022;Valanarasu et al 2023;Liu et al 2023) have been developed. For example, (Cun and Pun 2020;Hao, Iizuka, and Fukui 2020;Sofiiuk, Popenova, and Konushin 2021) proposed diverse attention modules to treat the foreground and background separately, or establish the relation between foreground and background.…”