Filling large holes in LiDAR data by inpainting depth gradients

Abstract

We introduce a technique to fill large holes in LiDAR data sets. We combine concepts from patch-based image inpainting and gradient-domain image editing to simultaneously fill both texture and structure in a LiDAR scan. We discuss the problems with directly inpainting a depth image, and present a solution to this problem based on inpainting the depth gradients. Once the inpainted depth gradients are obtained, we use an image reconstruction technique to obtain the final 3D scene structure. We present several realworld examples of this technique with excellent results.

Disocclusion of 3d Lidar Point Clouds Using Range Images

“…Patchbased methods such as (Criminisi et al, 2004) (and more recently (Buyssens et al, 2015b, Lorenzi et al, 2011) have proven their strengths. They have been extended for RGB-D images (Buyssens et al, 2015a) and to LiDAR point clouds (Doria and Radke, 2012) by considering an implicit topology in the point cloud. Variational approaches represent another type of inpainting algorithms (Chambolle and Pock, 2011, Bredies et al, 2010, Weickert, 1998, Bertalmio et al, 2000.…”

Building Scene Models by Completing and Hallucinating Depth and Semantics

“…In the context of stereo matching, Bleyer et al [35] introduce a method that hallucinates depth in the regions that are occluded in one view, but not in both. In [12,13], while the goal is indeed to replace the depth of foreground objects with that of the background, the methods assume to be given a perfect foreground mask, defined by a user. As a consequence, these approaches truly perform depth completion, albeit without the knowledge of the RGB intensity behind the foreground mask.…”

“…Typically, however, existing methods assume that the measurements are regularly spaced, and are thus ill-suited to handle large holes. By contrast, depth completion or inpainting [12,13] are designed to handle irregular measurements and fill holes in the input depth maps by leveraging RGB image information, or fusing multiple depth measurements [14]. These methods, however, simply complete the observed data.…”

“…Only little work has been done to tackle the task of depth hallucination from a noisy depth map and its corresponding RGB image [12,13,15,16], and existing methods typically work under additional assumptions. For example, [12,13] rely on a user-defined foreground mask to hallucinate the background depth.…”

“…For example, [12,13] rely on a user-defined foreground mask to hallucinate the background depth. The method in [15] relies on a layered depth model simply assuming that each layer is a smoothly varying surface, thus not considering semantics or image information.…”