Real-time smart stereo camera based on FPGA-SoC

Lan

et al. 2020

Sensors

Stereo vision utilizes two cameras to acquire two respective images, and then determines the depth map by calculating the disparity between two images. In general, object segmentation and stereo matching are some of the important technologies that are often used in establishing stereo vision systems. In this study, we implement a highly efficient self-organizing map (SOM) neural network hardware accelerator as unsupervised color segmentation for real-time stereo imaging. The stereo imaging system is established by pipelined, hierarchical architecture, which includes an SOM neural network module, a connected component labeling module, and a sum-of-absolute-difference-based stereo matching module. The experiment is conducted on a hardware resources-constrained embedded system. The performance of stereo imaging system is able to achieve 13.8 frames per second of 640 × 480 resolution color images.

Section: Resultsmentioning

confidence: 92%

Stereo Imaging Using Hardwired Self-Organizing Object Segmentation

Lan

et al. 2020

Sensors

“…Two competing approaches have been proposed when implementing rectification and undistortion algorithms. Some implement the necessary mathematical equations with arithmetic circuits [2] [4] [10], while others advocate for the use of lookup tables that perform similar functions to the equations [11] [12].…”

Section: Related Workmentioning

confidence: 99%

Model Based Design of a Real Time FPGA-Based Lens Undistortion and Image Rectification Algorithm for Stereo Imaging

Kaputa

Derhak

2023

IEEE Access

Low cost imaging sensors and powerful embedded computers have taken the field of computer vision to new heights. One of the challenges that remains is to shorten the development time that it takes to target one's algorithm to hardware. This work details the steps necessary to use Model Based Design to first simulate and then to target a stereo rectification and undistortion algorithm to an embedded FPGA system on chip hardware target. This work demonstrates what the future of FPGA-based algorithm development might look like by making use of closed loop design techniques with the aid of novel Model Based Design tool flows. Hardware-abstracted constructs such as 'virtual cameras' and the capability of pulling images off the target system and feeding them back into the simulation model help to minimize discrepancies between simulation and hardware domains, ultimately reducing the total development time. The developed system is a deterministic real time and low latency implementation as it does not rely on external off chip memory as all necessary buffering is done in the internal FPGA block RAM structures. The design is capable of processing 752x480 images at pixel line rates and is easily expandable to high resolution images. The algorithm being developed is not being touted as novel, rather that it is sufficiently complex in order to demonstrate the power of the Model Based Design tool flows.

“…NCC is even more complex to both SAD and SSD, because it involves many multiplication, division and square root operations [26]. The SAD method [27] is the most widely used matching technique in stereo vision algorithms, due to its low computational complexity, excellent performance and ease of hardware implementation.…”

Section: 𝑆𝐴𝐷 (𝑥 𝑦 𝑑) = ∑ |𝐼 𝐿 (𝑥 𝑦) − 𝐼 𝑅 (𝑥 𝑦 − 𝑑)| (𝑥𝑦)є𝑤mentioning

confidence: 99%

Parallel field programmable gate array implementation of the sum of absolute differences algorithm used in the stereoscopic system

et al. 2022

Stereo vision is a popular method for an artificial vision-based environment perception system used in various applications such as intelligent transportation. With two cameras, the disparity map is calculated to find the distance and depth of objects in front of a moving vehicle. The key element of the stereoscopic system is based on the sum of absolute differences (SAD) algorithm, which is the most repeated operation in the stereo matching subsystem; however, this algorithm requires a very intensive processing time, statistical analysis show that the SAD block can consume more than 80% of the overall processing time of the algorithm. In this paper we propose a highly efficient hardware architecture of the SAD algorithm for real time stereo matching, the proposed architecture is established by a hierarchical parallel architecture of the SAD block, and verified by simulation and successfully implemented in Cyclone IV field programmable gate array (FPGA), it provides a significant reduction of processing time and the performance of the stereo imaging system is able to achieve 30 frames per second of 640×480 resolution color images.