Low complexity optical flow using neighbor-guided semi-global matching

Abstract

Many real-time vision applications require accurate estimation of optical flow. This problem is quite challenging due to extremely high computation and memory requirements.This thesis focuses on designing low complexity dense optical flow algorithms.First, a new method for optical flow that is based on Semi-Global Matching (SGM), a popular dynamic programming algorithm for stereo vision, is presented. In SGM, the disparity of each pixel is calculated by aggregating local matching costs over the entire image to resolve local ambiguity in texture-less and occluded regions. The proposed method, Neighbor-Guided Semi-Global Matching (NG-fSGM) achieves significantly less complexity compared to SGM, by 1) operating on a subset of the search space that has been aggressively pruned based on neighboring pixels' information, 2) using a simple cost aggregation function, 3) approximating aggregated cost array and embedding pixel-wise matching cost computation and flow computation in aggregation. Evaluation on the Middlebury benchmark suite showed that, compared to a prior SGM extension for optical flow, the proposed basic NG-fSGM provides robust optical flow with 0.53% accuracy improvement, 40x reduction in number of operations and 6x reduction in memory size. To further reduce the complexity, sparse-to-dense flow estimation method is proposed. The number of operations and memory size are reduced by 68% and 47%, respectively, with only 0.42% accuracy degradation, compared to the basic NG-fSGM.A parallel block-based version of NG-fSGM is also proposed. The image is divided into overlapping blocks and the blocks are processed in parallel to improve throughput, latency ii and power efficiency. To minimize the amount of overlap among blocks with minimal effect on the accuracy, temporal information is used to estimate a flow map that guides flow vector selections for pixels along block boundaries. The proposed block-based NGfSGM achieves significant reduction in complexity with only 0.51% accuracy degradation compared to the basic NG-fSGM.

The Ames Stereo Pipeline: NASA's Open Source Software for Deriving and Processing Terrain Data

“…The version of the algorithm implemented by ASP has a few modifications relative to the original implementation. The most significant difference is that ASP's implementation performs a 2‐D disparity search, similar to what is done in the Neighbor‐Guided Semi Global Matching algorithm (Xiang et al, ). Since ASP processes a wide variety of cameras with varying degrees of metadata quality, the standard assumption with SGM that the disparity search can be performed only along a one‐dimensional epipolar line does not hold.…”

Underwater Target Detection and 3D Reconstruction System Based on Binocular Vision

“…Semi global matching (SGM) algorithm is known as a trade-off between accuracy and efficiency [ 31 , 32 ]. SGM methods adopt multiple paths optimization of disparity and achieve a minimum matching cost by the means of a winner-takes-all strategy based on hierarchical mutual information [ 33 , 34 ], which not only improves the calculation speed but also effectively solves the mismatch problem caused by the uneven illumination in images [ 35 ]. Therefore, it is a compromise strategy which is suitable for a real-time dense disparity map acquisition system based on binocular vision.…”

Three-dimensional (3D) reconstructions of the coastal cliff face in Normandy (France) based on oblique Pléiades imagery: assessment of Ames Stereo Pipeline® (ASP®) and MicMac® processing chains

“… Semi-Global Matching (SGM), introduced in Hirschmuller (2008). The "classical" SGM algorithm has undergone two important changes in ASP ® in order to include unrectified, larger images (NASA 2019): i) two-dimensional (2D) disparity search is performed, similarly to what is done in the Neighbor-Guided Semi-Global Matching algorithm (Xiang et al 2016) and ii) ASP ® uses a multi-resolution hierarchical search combined with a compressed memory scheme similar to what is used in the SGM algorithm (Rothermel et al 2012).…”