Deep Fusion Network for Image Completion

Abstract

Figure 1: Comparison results between DFNet and previous state-of-the-art method Edge Connect [21]. In the first image of each group, white pixels represent the unknown region. With fusion blocks along with multi-scale constraints, DFNet has smoother transition (1st case), more natural texture (2nd case) and more consistent structure (3rd case). AbstractDeep image completion usually fails to harmonically blend the restored image into existing content, especially in the boundary area. This paper handles with this problem from a new perspective of creating a smooth transition and proposes a concise Deep Fusion Network (DFNet). Firstly, a fusion block is introduced to generate a flexible alpha composition map for combining known and unknown regions. The fusion block not only provides a smooth fusion between restored and existing content, but also provides an attention map to make network focus more on the unknown pixels. In this way, it builds a bridge for structural and texture information, so that information can be naturally propagated from known region into completion. * This work is done when Xin Hong is an intern at Megvii Technology.Furthermore, fusion blocks are embedded into several decoder layers of the network. Accompanied by the adjustable loss constraints on each layer, more accurate structure information are achieved. We qualitatively and quantitatively compare our method with other state-of-the-art methods on Places2 and CelebA datasets. The results show the superior performance of DFNet, especially in the aspects of harmonious texture transition, texture detail and semantic structural consistency. Our source code will be avaiable at: https://github.com/hughplay/DFNet

Parallel Image Completion with Edge and Color Map

“…The deconvolution operations in GMCNN lead to the structure and color distortion. Although DFNet [38] shows a fine performance, it shows a lack of relevance between the hole and background regions such as symmetry of eyes, as shown in Figures 5e and 6f. The images completed by the proposed method PIC-EC and the state-of-the-art method EdgeConnect [23] are closer to the ground truth than images from other methods.…”

“…Yi et al [37] propose the GMCNN (Generative Multi-column Convolutional Neural Networks) model to synthesize different image components in a parallel manner within one stage. Xin et al [38] use a fusion block to generate a flexible alpha composition map providing a smooth fusion and an attention map to make the network focus more on the unknown regions.…”

Image Inpainting by Adaptive Fusion of Variable Spline Interpolations

“…In [13], a model called Deep Fusion Network (DFNet) has been proposed to address nonharmonic region boundaries. DFNet is an U-Net architecture embedded with introduced fusion blocks which applied in a multi-scale fashion.…”

Raindrop Removal With Light Field Image Using Image Inpainting

“…Different from the single image raindrop removal methods, which have the limitations on generalization capabilities, the image pairs, which are used to train the image inpainting task in a supervised learning paradigm, can be easily generated. It makes deep learning based image inpainting methods [10], [11] have million-level training datasets and work well even on the non-homologous images with pre-trained models. We therefore believe that the combination of light field image and image inpainting is helpful for raindrop removal task in general scenarios.…”

“…Nazeri et al [10] proposed a two-stage adversarial model that synthesizes edges of missing regions by an edge generator, then inputs predicted edges and incomplete color image to another generator for final inpainting. Moreover, Hong et al [11] recently achieved good performance of image completion using a U-Net architecture embedded with fusion blocks to the last few decoder layers.…”