Improving the Harmony of the Composite Image by Spatial-Separated Attention Module

Bargainnet: Background-Guided Domain Translation for Image Harmonization

“…Both traditional methods [4,5] and deep learning based methods [20,1,2,3] are included for quantitative comparisons.…”

“…However, they are more like style transfer and different from the photo-realistic harmonization in our task. More related to our work, in [1,2,3], they directly learn a mapping from composite images to real images, with the assistance of auxiliary semantic parsing branch [1], inserted attention models [2], or domain verification discriminator [3]. Different from these existing methods, our proposed method provides a new perspective by treating image harmonization as a background-guided domain translation.…”

“…Inspired by domain translation methods [25,9], we spatially replicate the L-dimensional domain code z b to an H × W × L domain code map Z b and concatenate it with the H × W × 3 composite image. Besides, based on our experimental observation (see Section 4.3 and Supplementary), it is still necessary to use foreground mask to indicate the foreground region to be harmonized as in [1,2,3], probably because the foreground mask emphasizes foreground translation and enables the foreground to borrow information from the background. Thus, we further concatenate the input with the H × W × 1 foreground mask M , leading to the final H × W × (L + 4) input.…”

“…Therefore, making the generated composite realistic could be a challenging task. Image harmonization [1,2,3], which aims to adjust the foreground to make it compatible with the background, is essential to address this problem. Traditional harmonization methods [4,5,6] improve the quality of synthesized composite mainly by transferring hand-crafted appearance statistics between foreground and background regions, but they could not handle the large appearance gap between foreground and background regions.…”

“…In [1], they presented the first end-to-end network for image harmonization. In [2], the spatial-separated attention blocks were proposed to learn the foreground and background features separately. Later in [3], they proposed an ad-versarial network with a domain verification discriminator to pull close the domains of foreground and background regions.…”