Joint segmentation of color and depth data based on splitting and merging driven by surface fitting

Abstract

This paper proposes a segmentation scheme based on the joint usage of color and depth data together with a 3D surface estimation scheme. Firstly a set of multi-dimensional vectors is built from color, geometry and surface orientation information. Normalized cuts spectral clustering is then applied in order to recursively segment the scene in two parts thus obtaining an over-segmentation. This procedure is followed by a recursive merging stage where close segments belonging to the same object are joined together. At each step of both procedures a NURBS model is fitted on the computed segments and the accuracy of the fitting is used as a measure of the plausibility that a segment represents a single surface or object. By comparing the accuracy to the one at the previous step it is possible to determine if each splitting or merging operation leads to a better scene representation and consequently whether to perform it or not. Experimental results show how the proposed method provides an accurate and reliable segmentation.

Guided optimization framework for the fusion of time-of-flight with stereo depth

“…19 They re-enforce the similarities by edge-weighting schemes to achieve a better estimation of the fine details. Some approaches 3,20 came up with the idea of guiding the optimization framework by applying image segmentation that results in better edge recovery. Other approaches [21][22][23] formulate their guidance as an edge-weighting scheme extracted by enforcing spatial similarity between neighboring pixels from both image and depth data without explicit image segmentation.…”