Guidance and Evaluation: Semantic-Aware Image Inpainting for Mixed Scenes

Liao, Liang; Xiao, Jing; Wang, Zheng; Lin, Chia‐Wen; Satoh, Shin’ichi

doi:10.1007/978-3-030-58583-9_41

Cited by 89 publications

(43 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, the area with the same semantic labels may have different textures in image inpainting(e.g., continuous but different windows), thus the same label will mislead the process of inpainting in these areas. And then, Liao et al [36] propose a selfevaluation mechanism for image inpainting through segmentation confidence scoring to localize the predicted pixels in the supervised dataset. Recently, some researchers utilize explicit image structure knowledge for inpainting.…”

Section: B Image Inpainting By Deep Generative Modelsmentioning

confidence: 99%

Semantic-SCA: Semantic Structure Image Inpainting With the Spatial-Channel Attention

2021

View full text Add to dashboard Cite

Deep learning has brought unprecedented progress to image inpainting. However, the existing methods often generate images with blurry textures and distorted structures because they may either fail to maintain semantic consistency or restore fine-grained textures. In this paper, we propose a two-stage adversarial model to further improve the accuracy of the structure and details of image inpainting. Our model splits the inpainting task into two parts: semantic structure reconstructor and texture generator. In the first stage, we first utilize the semantic structure map based on the unsupervised segmentation to train the semantic structure reconstructor, which completes the missing structures of the inputs and maintains consistency between the missing part and the overall image. In the second stage, we introduce the spatial-channel attention (SCA) module to obtain the fine-grained textures. The SCA module strengthens the capability to obtain information from the long-distance pixel and different channels of the model. Furthermore, we propose a spatial-channel loss to stabilize the network training process and improve visual effects. Finally, we evaluate our model over the publicly available datasets CelebA, Places2, and Paris StreetView. When the inpainting tasks involved in large-area defects or heavy structure, the experimental results show that our method has a higher inpainting quality than the existing state-of-the-art approaches. INDEX TERMS Artificial neural networks, deep learning, generative model, image generation, image inpainting.

show abstract

Section: B Image Inpainting By Deep Generative Modelsmentioning

confidence: 99%

Semantic-SCA: Semantic Structure Image Inpainting With the Spatial-Channel Attention

2021

View full text Add to dashboard Cite

show abstract

“…Song et al [6] first take semantic information into the modeling of texture. Liao et al [7] propose SGE-Net to iteratively update the structural priors and inpainting image at the same time. However, how to extract complete image priors and use them properly remains challenging, especially for the complex scene.…”

Section: Image Prior Guided Inpaintingmentioning

confidence: 99%

“…2019-A1515011075). inpainting methods introduce mid-level structural information to generate semantically correct results, such as edges [5], segmentations [6] [7], or features directly [8]. EC [5] and SPG [6] disentangle the inpainting task into structural information completing and structural information guided image inpainting.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Structural Prior Guided Image Inpainting for Complex Scene

Wei

Gao

2021

2021 IEEE International Conference on Multimedia and Expo (ICME)

View full text Add to dashboard Cite

One of the main challenges faced by many existing image inpainting methods is that they fail to generate clear boundaries between different objects in a complex scene. To address this problem, we propose a novel two-stage image completion framework with semantic segmentation maps as the structure prior to restrain texture generation. Our framework consists of a PSP-based adversarial network called SP-Net, which can generate a semantic segmentation map for the corrupted image and an SG-Net using domain alignment and correlation matrix to get the feature correspondence between structural prior and known region. Combining the structural prior and feature correspondence, the image completion network fills the missing region with distinct boundaries and fine details. The proposed method was evaluated over Outdoor Scenes and Cityscapes datasets. Experimental results demonstrate that our approach outperforms other structural prior based approaches in terms of visual quality and quantitative measurements.

show abstract

“…It is worth noting that, different from some methods [14,15] that generate the semantic map to assist the image inpainting under the supervision of ground-truth labels, SACA predicts the semantic masks to conduct the context aggregation by exploiting internal semantic similarity of the input in an unsupervised way, so it can be applied in more scenarios.…”

Section: Introductionmentioning

confidence: 99%

Semantic-Aware Context Aggregation for Image Inpainting

Huang

Qin

Liu

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Recent attention-based image inpainting methods have made inspiring progress by propagating distant contextual information into holes. However, they tend to generate blurry contents since the propagation process is always misled by preliminarily-recovered holes features which are not well-inferred. To handle this problem, we propose a novel semantic-aware context aggregation module (SACA) that aggregates distant contextual information from a semantic perspective by exploiting the internal semantic similarity of the input feature map. Compared with existing attention mechanisms that model the relation of all pixel-pairs, SACA can suppress the impact of misleading holes features in context aggregation and significantly reduce computation burden by learning the relation between pixels and semantics. Also, we apply SACA to both high-level and low-level feature maps in our model for generating both semantically and visually plausible results. Extensive experiments on Outdoor Scenes, CelebA and Paris StreetView datasets validate the superiority of our method compared with existing methods.

show abstract

Guidance and Evaluation: Semantic-Aware Image Inpainting for Mixed Scenes

Cited by 89 publications

References 39 publications

Semantic-SCA: Semantic Structure Image Inpainting With the Spatial-Channel Attention

Semantic-SCA: Semantic Structure Image Inpainting With the Spatial-Channel Attention

Structural Prior Guided Image Inpainting for Complex Scene

Semantic-Aware Context Aggregation for Image Inpainting

Contact Info

Product

Resources

About