Object removal by exemplar-based inpainting

Abstract

A new algorithm is proposed for removing large objects from digital images. The challenge is to fill in the hole that is left behind in a visually plausible way.In the past, this problem has been addressed by two classes of algorithms: (i) "texture synthesis" algorithms for generating large image regions from sample textures, and (ii) "inpainting" techniques for filling in small image gaps. The former work well for "textures" -repeating twodimensional patterns with some stochasticity; the latter focus on linear "structures" which can be thought of as onedimensional patterns, such as lines and object contours.This paper presents a novel and efficient algorithm that combines the advantages of these two approaches. We first note that exemplar-based texture synthesis contains the essential process required to replicate both texture and structure; the success of structure propagation, however, is highly dependent on the order in which the filling proceeds. We propose a best-first algorithm in which the confidence in the synthesized pixel values is propagated in a manner similar to the propagation of information in inpainting. The actual colour values are computed using exemplar-based synthesis. Computational efficiency is achieved by a blockbased sampling process.A number of examples on real and synthetic images demonstrate the effectiveness of our algorithm in removing large occluding objects as well as thin scratches. Robustness with respect to the shape of the manually selected target region is also demonstrated. Our results compare favorably to those obtained by existing techniques.

Building Scene Models by Completing and Hallucinating Depth and Semantics

“…In Fg-Mask, however, no appearance information about the background is available. We therefore apply the technique of [28] to inpaint this area, which, to the best of our knowledge, remains the most mature method when it comes to depth completion without intensity information. This yields two baselines, which we will refer to as Baseline-1 (semantic segmentation followed by [29] + [28]) and Baseline-2 (semantic segmentation followed by [27] + [28]).…”

“…We therefore apply the technique of [28] to inpaint this area, which, to the best of our knowledge, remains the most mature method when it comes to depth completion without intensity information. This yields two baselines, which we will refer to as Baseline-1 (semantic segmentation followed by [29] + [28]) and Baseline-2 (semantic segmentation followed by [27] + [28]). To compare the different algorithms, we make use of the following metrics:1) visible-rmse: the-root-mean-square-error (rmse) for the entire depth map; 2)hidden-rmse: the rmse for the depth map hallucinated underneath the ground truth foreground mask.…”

“…In [27], depth completion is formulated within a total variation framework where image cues guide the completion process. A different approach to depth completion consists of treating a depth map as an intensity image, and rely on standard image inpainting algorithms, such as [28] and [29]. All the above-mentioned methods focus on depth completion form a single view and aim at completing the visible scene information only.By contrast, some approaches have proposed to exploit multiple views [30,14] and thus can handle the fact that parts of the scene are hidden in some views, albeit not all of them.…”

Repairing Sparse Low-Rank Texture

“…More specifically, it requires that the pixel values of the missing regions follow the same statistical or geometric structures as the rest of the image. In the literature, based on different assumptions on the (local or global) statistics or structures of the input image, many methods which can directly or indirectly deal with image completion problems have been developed for different types of images or textures: from the synthesis of stationary stochastic random textures [1][2][3], to the inpainting of piece-wise smooth natural images [4][5][6] , and to the synthesis of highly symmetric and highly-structured regular textures [7]. In this paper, we focus on the class of images or textures whose structures have very low intrinsic dimensionality or complexity.…”

Rotationally invariant similarity measures for nonlocal image denoising

“…Exemplarbased algorithms, e.g. [13], fill in missing image data or remove certain image objects by searching for similar regions or structures in the image and completing the missing data or overwriting the data to be removed according to the information found there. A very popular denoising approach that is motivated by the inpainting method of Efros and Leung [15] is the so-called nonlocal means (NL means) algorithm for image denoising that has been proposed by Buades et al [7,8].…”