A Novel Factor Graph-Based Optimization Technique for Stereo Correspondence Estimation

Abstract

Dense disparities among multiple views is essential for estimating the 3D architecture of a scene based on the geometrical relationship among the scene and the views or cameras. Scenes with larger extents of heterogeneous textures, differing scene illumination among the multiple views and with occluding objects affect the accuracy of the estimated disparities. Markov random fields (MRF) based methods for disparity estimation address these limitations using spatial dependencies among the observations and among the disparity estimates. These methods, however, are limited by spatially fixed and smaller neighborhood systems or cliques. In this work, we present a new factor graph-based probabilistic graphical model for disparity estimation that allows a larger and a spatially variable neighborhood structure determined based on the local scene characteristics. We evaluated our method using the Middlebury benchmark stereo datasets and the Middlebury evaluation dataset version 3.0 and compared its performance with recent state-of-the-art disparity estimation algorithms. The new factor graph-based method provided disparity estimates with higher accuracy when compared to the recent non-learning-and learning-based disparity estimation algorithms. In addition to disparity estimation, our factor graph formulation can be useful for obtaining maximum a posteriori solution to optimization problems with complex and variable dependency structures as well as for other dense estimation problems such as optical flow estimation.

Multi-Resolution Factor Graph Based Stereo Correspondence Algorithm

“…and a set of factor nodes from multiple resolutions F = within each level ζ using local image characteristics as in the FGS model [10]. In brief, an αth percentile cut-off of the bilateral filter [29] coefficients estimated at the ith pixel location were used to identify neighboring variable nodes that have the highest influence on the true state of the disparity d i .…”

“…Probability of assigning various disparity labels to each pixel i at any given multi-resolution level ζ can be obtained by marginalizing equation ( 1) with respect to D \ i as [10,30,31]. For an approximate and efficient computation of marginal beliefs or probabilities in equation ( 2) using loopy belief propagation, local information available in each node is shared with neighboring nodes as variable-to-dependency factor messages µ i∈V→f ∈(S∪R) and factor-to-variable messages µ f ∈F→i∈V until convergence [32,33].…”

“…The MR-FGS algorithm follows the same message passing structure as the FGS algorithm [10]. In brief, the variable-to-dependency factor node message µ i∈V→f ∈(S∪R) in equation ( 3) approximated the posterior probability of d i while satisfying all of the constraints from its neighboring spatial and resolution dependency factor nodes except the factor node f to which the message is sent.…”

“…Similarly, the factor node-to-variable node messages µ f ∈F→i∈V in equation ( 4) approximated the likelihood of satisfying spatial or resolution dependencies among the random variable states of all variables nodes associated with the factor node f except d i . Relationship between the probabilistic solution structure of the MR-FGS algorithm with message passing is same as that of the FGS algorithm [10]. Thus, the MR-FGS algorithm determines optimal disparities based on approximate inference of the posterior probability of assigning various disparity labels at each pixel i p…”

“…Though MRF-based stereo matching algorithms improve the accuracy of the disparity estimates, they require maximal spatial dependencies among pixels in the chosen MRF neighborhood system. Further, because the neighborhood system is uniformly enforced for all the pixel locations, pairwise cliques or factor graph-based probabilistic graphical model (FGS) that addresses these limitations of MRF-based disparity optimization techniques [10]. More specifically, the FGS method allows spatially variable, larger and computationally optimal neighborhood systems for probabilistic graphical models.…”