StructureFlow: Image Inpainting via Structure-Aware Appearance Flow

Abstract

Image inpainting techniques have shown significant improvements by using deep neural networks recently. However, most of them may either fail to reconstruct reasonable structures or restore fine-grained textures. In order to solve this problem, in this paper, we propose a two-stage model which splits the inpainting task into two parts: structure reconstruction and texture generation. In the first stage, edgepreserved smooth images are employed to train a structure reconstructor which completes the missing structures of the inputs. In the second stage, based on the reconstructed structures, a texture generator using appearance flow is designed to yield image details. Experiments on multiple publicly available datasets show the superior performance of the proposed network.

Image Inpainting Using Two-Stage Loss Function and Global and Local Markovian Discriminators

“…Song et al [18] proposed segmentation prediction and guidance network and took the image after semantic segmentation as the intermediate state of two-stage network. Ren et al [19] first recovered the structural image by removing the high-frequency components and then used the appearance flow to obtain the final result.…”

“…However, these methods may fail to effectively separate structure and texture information, so the results tend to be boundary crossing smooth or texture distortion. To solve this problem, some image inpainting approaches [15][16][17][18][19] were proposed using the two-stage networks. These methods restore the proper image structure in stage 1, and then use the reconstructed information in stage 2 to generate the final result.…”

“…These methods restore the proper image structure in stage 1, and then use the reconstructed information in stage 2 to generate the final result. However, these methods [18,19] are usually time-consuming owning to the pre-processing of dataset before training. Besides, two stages of networks [17][18][19] usually need to be trained separately; thus, the end-to-end training cannot be achieved and the training complexity is increased.…”

“…However, these methods [18,19] are usually time-consuming owning to the pre-processing of dataset before training. Besides, two stages of networks [17][18][19] usually need to be trained separately; thus, the end-to-end training cannot be achieved and the training complexity is increased. In addition, some methods may fail to select an appropriate intermediate state, which will result in the misleading or less available information in state 2 of network.…”

Image Inpainting Based on Multi-frequency Probabilistic Inference Model

“…To solve this problem, some two-stage networks [18,23,25,36] are proposed to recover rough structure of missing area in the first stage and generate final results using the reconstructed information in the second stage. However, the second generation network largely depends on the correct results of the first reconstruct network and the two-step sequential training also brings additional computing burden.…”