Hardware-Efficient Neighbor-Guided SGM Optical Flow for Low Power Vision Applications

Xiang, Jiang; Li, Ziyun; Kim, Hun-Seok; Chakrabarti, Chaitali

doi:10.1109/sips.2016.8

Cited by 6 publications

(3 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…System-level Optimizations While many have studied accelerators for individual tasks in localization, such as convolution [72], stereo matching [33], [57], and optical flow [40], [84], an end-to-end localization system necessarily integrates different tasks, and requires us to explore optimizations at the system level [47]. Our hardware design exploits the parallelism and data communication patterns across different tasks, and judiciously applies pipelining, resource-sharing, on-chip buffer specialization to achieve meaningful system-level gains.…”

Section: Discussionmentioning

confidence: 99%

Eudoxus: Characterizing and Accelerating Localization in Autonomous Machines

Gan,

Yu,

Tian

et al. 2020

Preprint

View full text Add to dashboard Cite

We develop and commercialize autonomous machines, such as logistic robots and self-driving cars, around the globe. A critical challenge to our-and any-autonomous machine is accurate and efficient localization under resource constraints, which has fueled specialized localization accelerators recently. Prior acceleration efforts are point solutions in that they each specialize for a specific localization algorithm. In realworld commercial deployments, however, autonomous machines routinely operate under different environments and no single localization algorithm fits all the environments. Simply stacking together point solutions not only leads to cost and power budget overrun, but also results in an overly complicated software stack.This paper demonstrates our new software-hardware codesigned framework for autonomous machine localization, which adapts to different operating scenarios by fusing fundamental algorithmic primitives. Through characterizing the software framework, we identify ideal acceleration candidates that contribute significantly to the end-to-end latency and/or latency variation. We show how to co-design a hardware accelerator to systematically exploit the parallelisms, locality, and common building blocks inherent in the localization framework. We build, deploy, and evaluate an FPGA prototype on our next-generation selfdriving cars. To demonstrate the flexibility of our framework, we also instantiate another FPGA prototype targeting drones, which represent mobile autonomous machines. We achieve about 2× speedup and 4× energy reduction compared to widely-deployed, optimized implementations on general-purpose platforms.

show abstract

Section: Discussionmentioning

confidence: 99%

Eudoxus: Characterizing and Accelerating Localization in Autonomous Machines

Gan,

Yu,

Tian

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…An earlier version of the work appeared in [12]. The image is divided into overlapping blocks and the blocks are processed in parallel to improve throughput, latency and power efficiency.…”

Section: Contributionmentioning

confidence: 99%

“…An earlier version of this work appeared in [12]. Recall that the memory size and number of operations in NG-fSGM is a function of the image size.…”

Section: Chapter 4 Parallel Block-based Ng-fsgmmentioning

confidence: 99%

Low complexity optical flow using neighbor-guided semi-global matching

Xiang

Blaauw

et al. 2016

2016 IEEE International Conference on Image Processing (ICIP)

Self Cite

View full text Add to dashboard Cite

Many real-time vision applications require accurate estimation of optical flow. This problem is quite challenging due to extremely high computation and memory requirements.This thesis focuses on designing low complexity dense optical flow algorithms.First, a new method for optical flow that is based on Semi-Global Matching (SGM), a popular dynamic programming algorithm for stereo vision, is presented. In SGM, the disparity of each pixel is calculated by aggregating local matching costs over the entire image to resolve local ambiguity in texture-less and occluded regions. The proposed method, Neighbor-Guided Semi-Global Matching (NG-fSGM) achieves significantly less complexity compared to SGM, by 1) operating on a subset of the search space that has been aggressively pruned based on neighboring pixels' information, 2) using a simple cost aggregation function, 3) approximating aggregated cost array and embedding pixel-wise matching cost computation and flow computation in aggregation. Evaluation on the Middlebury benchmark suite showed that, compared to a prior SGM extension for optical flow, the proposed basic NG-fSGM provides robust optical flow with 0.53% accuracy improvement, 40x reduction in number of operations and 6x reduction in memory size. To further reduce the complexity, sparse-to-dense flow estimation method is proposed. The number of operations and memory size are reduced by 68% and 47%, respectively, with only 0.42% accuracy degradation, compared to the basic NG-fSGM.A parallel block-based version of NG-fSGM is also proposed. The image is divided into overlapping blocks and the blocks are processed in parallel to improve throughput, latency ii and power efficiency. To minimize the amount of overlap among blocks with minimal effect on the accuracy, temporal information is used to estimate a flow map that guides flow vector selections for pixels along block boundaries. The proposed block-based NGfSGM achieves significant reduction in complexity with only 0.51% accuracy degradation compared to the basic NG-fSGM.

show abstract