Discrete optimization of ray potentials for semantic 3D reconstruction

Abstract

Dense semantic 3D reconstruction is typically formulated as a discrete or continuous problem over label assignments in a voxel grid, combining semantic and depth likelihoods in a Markov Random Field framework. The depth and semantic information is incorporated as a unary potential, smoothed by a pairwise regularizer. However, modelling likelihoods as a unary potential does not model the problem correctly leading to various undesirable visibility artifacts.We propose to formulate an optimization problem that directly optimizes the reprojection error of the 3D model with respect to the image estimates, which corresponds to the optimization over rays, where the cost function depends on the semantic class and depth of the first occupied voxel along the ray. The 2-label formulation is made feasible by transforming it into a graph-representable form under QPBO relaxation, solvable using graph cut. The multi-label problem is solved by applying α-expansion using the same relaxation in each expansion move. Our method was indeed shown to be feasible in practice, running comparably fast to the competing methods, while not suffering from ray potential approximation artifacts.

RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials

“…However, their method lacks a global probabilistic formulation. To address this limitation, a number of approaches have phrased 3D reconstruction as inference in a Markov random field (MRF) by exploiting the special characteristics of high-order ray potentials [19,28,33,35]. Ray potentials allow for accurately describing the image formation process, yielding 3D reconstructions consistent with the input images.…”

“…Our MRF aggregates these depth distributions by exploiting occlusion relationships across all viewpoints, yielding significantly improved depth predictions. Occlusion constraints are encoded using high-order ray potentials [19,28,35]. We differ from [19,35] in that we do not reason about surface appearance within the MRF but instead incorporate depth distributions estimated by our CNN.…”

“…While this formulation correctly models occlusion, the complex nature of inference in ray potentials restricts these models to pixel-wise color comparisons, which leads to large ambiguities in the reconstruction [35]. Instead of using images as input, Savinov et al [28] utilize pre-computed depth maps using zero-mean normalized cross-correlation in a small image neighborhood. In this case, the ray potentials encourage consistency between the input depth map and the depth of the first visible voxel along the ray.…”

Patchwork Stereo: Scalable, Structure-Aware 3D Reconstruction in Man-Made Environments

“…Despite mesh refinement, the resulting surface remains a jagged approximation of a locally-smooth geometry, which may then require expensive post-processing to achieve a compact representation, e.g., by fitting 3D geometric primitives [19,15]. The situation is even worse with voxel-based approaches [12,30]. Pixel-based stereo techniques, which build disparity maps, have seen a tremendous increase in performance since early approaches [16] and their later extensions using second order smoothness priors [40,27], color models [2] or semantic classification [18].…”

A Data-Driven Prior on Facet Orientation for Semantic Mesh Labeling

“…The pairwise term is the combination of smoothness and class-specific priors, the latter is learned from manually annotated data, and defines the probability of class changes depending on the normal transition. This work has been extended in [29,28] to embed higher order ray potentials into the optimization of the graphical model. For a more in-depth discussion about semantic volumetric reconstruction we refer the reader to [15].…”