Non-Local Patch-Based Image Inpainting

Abstract

Image inpainting is the process of filling in missing regions in an image in a plausible way. In this contribution, we propose and describe an implementation of a patch-based image inpainting algorithm. The method is actually a two-dimensional version of our video inpainting algorithm The functional specifies that a good solution to the inpainting problem should be an image where each patch is very similar to its nearest neighbor in the unoccluded area. Iterations are performed in a multi-scale framework which yields globally coherent results. In this manner two of the major goals of image inpainting, the correct reconstruction of textures and structures, are addressed. We address a series of important practical issues which arise when using such an approach. In particular, we reduce execution times by using the PatchMatch [C. Barnes, PatchMatch: a randomized correspondence algorithm for structural image editing, ACM Transactions on Graphics, (2009)] algorithm for nearest neighbor searches, and we propose a modified patch distance which improves the comparison of textured patches. We address the crucial issue of initialization and the choice of the number of pyramid levels, two points which are rarely discussed in such approaches. We provide several examples which illustrate the advantages of our algorithm, and compare our results with those of state-of-the-art methods. Source CodeThe reviewed source code and documentation for this algorithm are available from the web page of this article 1 .

Onion-Peel Networks for Deep Video Completion

“…-Our framework is applicable to the image completion guided by the reference images that is hard to be achieved by the previous image and video completion methods. -We validate that our method produces comparable results 2 The term "onion peel" originated from the initialization technique in [23,24] for the randomized search through the PatchMatch [1] to the state-of-the-art methods with fast computational time.…”

Object removal from complex videos using a few annotations

“…• Unlike OSVOS, which uses a fully convolutional network (FCN) [73], our network uses the Deeplab v2 [74] architecture as the parent model since it outperforms FCN on some common datasets such as PASCAL VOC 2012 [75]. • In the fine-tuning training step we adopt a data augmentation technique in the spirit of Lucid Tracker [21]: we remove all objects from the first frame using Newson et al's image inpainting algorithm [76], then the removed objects undergo random geometric deformations (affine and thin plate deformations), and eventually are Poisson blended [77] over the reconstructed background. This is a sensible way of generating large amounts of labeled training data with an appearance similar to that which the network might observe in the following frames.…”

High-Resolution Image Inpainting Based on Multi-Scale Neural Network

“…Traditional inpainting approaches based on diffusion or patch typically use variational algorithms or patch similarity to spread information from background to holes, such as [23,24]. One of the most advanced methods for image inpainting at present is PatchMatch [25], without the use of deep learning, which fills in holes with statistical data of available images through iterated search for the most suitable patch.…”