Back to Butterworth - a Fourier basis for 3D surface relief hole filling within RGB-D imagery

Abstract

We address the problem of hole filling in RGB-D (color and depth) images, obtained from either active or stereo based sensing, for the purposes of object removal and missing depth estimation. This is performed independently on the low frequency depth information (surface shape) and the high frequency depth detail (relief) by way of a Fourier space transform and classical Butterworth high/low pass filtering. The high frequency detail is then filled using a texture synthesis method, whilst the low frequency shape information is inpainted using structural inpainting. Here, a classical non-parametric sampling approach is extended, using the concept of query expansion, to perform high frequency depth synthesis with the final output then recombined in Fourier space. In order to improve the overall depth relief (D) and edge detail accuracy, color information (RGB) is also used to constrain the sampling process within high frequency component completion. Experimental results demonstrate the efficacy of the proposed method outperforming prior work for generalized depth filling in the presence of high frequency surface relief detail.

Veritatem Dies Aperit - Temporally Consistent Depth Prediction Enabled by a Multi-Task Geometric and Semantic Scene Understanding Approach

“…The effective performance of our segmentation points to the generalization capabilities of our model. When tested on CamVid [14], our approach produces better results compared to well-established tech- [9]; GLC: Global and Local Completion [36]; ICA: Inpainting with Contextual Attention [82]; GIF: Guided Inpainting and Filtering [50].…”

“…Figure 17: Comparison of depth completion methods applied to synthetic test set. RGB: input colour image; GTD: Ground Truth Depth; DH: Depth Holes; FDF: Fourier based Depth Filling [9]; GTS: Global and Local Completion [36]; ICA: Inpainting with Contextual Attention [82]; GIF: Guided Inpainting and Filtering [50].…”

“…• Temporal Continuity -temporal information is explicitly taken into account during training using both recurrent network feedback and gradients from a pre-trained frozen optical flow network. This leads to a novel scene understanding approach capable of temporally consistent geometric depth prediction and semantic scene segmentation whilst outperforming prior work across the domains of monocular depth estimation [8,25,29,49,83,87], completion [9,36,50,82] and semantic segmentation [10,17,40,52,53,59,74,75,86].…”

To Complete or to Estimate, That is the Question: A Multi-Task Approach to Depth Completion and Monocular Depth Estimation

“…However, pervasive issues and complications ever-present in depth-reliant vision systems (e.g. missing depth, temporal and in-scene consistency, intensive computational and calibration requirements and alike), have led to the emergence of entire areas of research focusing on refinement procedures post estimation or capture [3,8,10,16,39,55] to render scene depth more useful for downstream applications.…”

Extended Patch Prioritization for Depth Filling Within Constrained Exemplar-Based RGB-D Image Completion

“…Despite significant prior work in color image completion [1-3, 6, 8, 11], depth filling is by contrast scantly present within the literature [4,7,9,10,13,30] emerging as a relatively new research area posing significant challenges [12]. Although there have been many attempts to use structure-based or exemplar-based color image completion approaches for depth hole filling [1][2][3]5], particular factors such as the absence of granular texture, clear object separation and the lack of in-scene transferability of varying depth sub-regions all create notable obstacles not present in the corresponding color completion case [29].…”