Region Filling and Object Removal by Exemplar-Based Image Inpainting

Abstract

A new algorithm is proposed for removing large objects from digital images. The challenge is to fill in the hole that is left behind in a visually plausible way. In the past, this problem has been addressed by two classes of algorithms: 1) "texture synthesis" algorithms for generating large image regions from sample textures and 2) "inpainting" techniques for filling in small image gaps. The former has been demonstrated for "textures"--repeating two-dimensional patterns with some stochasticity; the latter focus on linear "structures" which can be thought of as one-dimensional patterns, such as lines and object contours. This paper presents a novel and efficient algorithm that combines the advantages of these two approaches. We first note that exemplar-based texture synthesis contains the essential process required to replicate both texture and structure; the success of structure propagation, however, is highly dependent on the order in which the filling proceeds. We propose a best-first algorithm in which the confidence in the synthesized pixel values is propagated in a manner similar to the propagation of information in inpainting. The actual color values are computed using exemplar-based synthesis. In this paper, the simultaneous propagation of texture and structure information is achieved by a single, efficient algorithm. Computational efficiency is achieved by a block-based sampling process. A number of examples on real and synthetic images demonstrate the effectiveness of our algorithm in removing large occluding objects, as well as thin scratches. Robustness with respect to the shape of the manually selected target region is also demonstrated. Our results compare favorably to those obtained by existing techniques.

Patch-based segmentation using expert priors: Application to hippocampus and ventricle segmentation

“…As exemplars, patch-based methods are currently the focus of attention of the computer vision community in various domains such as texture synthesis (Efros and Freeman, 2001), in-painting (Criminisi et al, 2004), restoration (Buades et al, 2005), and single-frame super resolution (Protter et al, 2009). In each of these domains, patch-based methods have been the subject of intensive investigation because they exhibit very high performance despite their simplicity.…”

Non-Local Sparse Image Inpainting for Document Bleed-Through Removal

“…In [20], a patch priority based inpainting is suggested that extends known image patches to the missing parts of the image. A non-local exemplar based method is suggested in [60], where the missing patches are estimated as means of selected non-local patches.…”

“…Image inpainting, which refers to filling in missing or corrupted regions in an image, is a well studied and challenging topic in computer vision and image processing [19,20]. In image inpainting, the goal is to find an estimate for those regions in order to reconstruct a visually pleasant and consistent image [21].…”

4. Rotation invariance in exemplar-based image inpainting

“…This concept was picked up and further analyzed Wei and Levoy [WL00] and justified in terms of probability theory by Levina and Bickel [LB06]. Criminisi et al applied the concept to inpainting problems like object removal in [CPT04].…”

“…The algorithms in [EL99,WL00] are greedy methods for the computation of a suitable correspondence map, since each pixel in the inpainting domain is visited only once. As it was shown in [CPT04] the quality of the result heavily depends on the order in which the pixels are processed. The two results in the Subfigures 1c and 1d are actually obtained by the same method of comparing patches, but just visiting the pixels in a different order.…”