Event-based Plane-fitting Optical Flow for Dynamic Vision Sensors in FPGA

Aung, Myo Tun; Teo, Rodney; Orchard, Garrick

doi:10.1109/iscas.2018.8351588

Cited by 20 publications

(26 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Firstly, adaptations of the gradient-based Lucas-Kanade algorithm [51] were presented in [52], [53]. Secondly, the methods proposed in [13], [54], [55] extract optical flow by computing the gradients of a local plane fitted to a spatiotemporal surface of events. Thirdly, multiple adaptations of the bio-inspired frequency-based methods have been introduced [53], [56], [57], which allow the implementation in neuromorphic hardware [58].…”

Section: Event-based Optical Flow Estimationmentioning

confidence: 99%

Unsupervised Learning of a Hierarchical Spiking Neural Network for Optical Flow Estimation: From Events to Global Motion Perception

Paredes-Vallés

Scheper

Croon

2020

IEEE Trans. Pattern Anal. Mach. Intell.

131

View full text Add to dashboard Cite

The combination of spiking neural networks and event-based vision sensors holds the potential of highly efficient and high-bandwidth optical flow estimation. This paper presents the first hierarchical spiking architecture in which motion (direction and speed) selectivity emerges in an unsupervised fashion from the raw stimuli generated with an event-based camera. A novel adaptive neuron model and stable spike-timing-dependent plasticity formulation are at the core of this neural network governing its spike-based processing and learning, respectively. After convergence, the neural architecture exhibits the main properties of biological visual motion systems, namely feature extraction and local and global motion perception. Convolutional layers with input synapses characterized by single and multiple transmission delays are employed for feature and local motion perception, respectively; while global motion selectivity emerges in a final fully-connected layer. The proposed solution is validated using synthetic and real event sequences. Along with this paper, we provide the cuSNN library, a framework that enables GPU-accelerated simulations of large-scale spiking neural networks. Source code and samples are available at https://github.com/tudelft/cuSNN. Index Terms-Event-based vision, feature extraction, motion detection, neural nets, neuromorphic computing, unsupervised learning ! • The authors are with the Department of Control and Simulation (Micro Air Vehicle

show abstract

Section: Event-based Optical Flow Estimationmentioning

confidence: 99%

Unsupervised Learning of a Hierarchical Spiking Neural Network for Optical Flow Estimation: From Events to Global Motion Perception

Paredes-Vallés

Scheper

Croon

2020

IEEE Trans. Pattern Anal. Mach. Intell.

131

View full text Add to dashboard Cite

show abstract

“…When compared to the estimated resource utilization in [8], the hARMS architecture offers configurations capable of achieving real-time performance with fewer resources. The achieved throughput of 1.21 Mevt/s enables hARMS to nearly match the throughput achieved by the local-flow FPGA implementation in [9]. Matching this performance means that the design in [9] could be used to generate the local-flow input to the hARMS architecture.…”

Section: Discussionmentioning

confidence: 86%

“…Another example of event-based optical flow acceleration using an FPGA is presented in [9]. This research presents an FPGA implementation for a modified version of the iterative derivative of the surface of events algorithm presented in [4].…”

Section: Introductionmentioning

confidence: 99%

“…The algorithm derives the surface of events after a temporal regularization to asynchronously estimate the flow, and is capable of performing at a throughput of 2.75 Mevt/s. However, the overall throughput of the system is limited to 1.46 Mevt/s due to the pre-processing stage [9]. The design was implemented using a Xilinx Spartan 6 LX150 FPGA on an Opal Kelly XEM6010 board and required 3794 logic slices, 138 block RAMs (BRAM), and 16 DSPs when using a 304×240 px resolution ATIS sensor.…”

Section: Introductionmentioning

confidence: 99%

“…While results presented in [9] show impressive throughput, allowing for real-time, asynchronous operation, it makes no attempt to address the aperture problem. The design presented also has high BRAM requirements because recent events at each pixel location are stored.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

hARMS: A Hardware Acceleration Architecture for Real-Time Event-Based Optical Flow

Stumpp¹,

Akolkar²,

George³

et al. 2021

Preprint

View full text Add to dashboard Cite

Event-based vision sensors produce asynchronous event streams with high temporal resolution based on changes in the visual scene. The properties of these sensors allow for accurate and fast calculation of optical flow as events are generated. Existing solutions for calculating optical flow from event data either fail to capture the true direction of motion due to the aperture problem, do not use the high temporal resolution of the sensor, or are too computationally expensive to be run in real time on embedded platforms. In this research, we first present a faster version of our previous algorithm, ARMS (Aperture Robust Multi-Scale flow). The new optimized software version (fARMS) significantly improves throughput on a traditional CPU. Further, we present hARMS, a hardware realization of the fARMS algorithm allowing for real-time computation of true flow on low-power, embedded platforms. The proposed hARMS architecture targets hybrid system-on-chip devices and was designed to maximize configurability and throughput. The hardware architecture and fARMS algorithm were developed with asynchronous neuromorphic processing in mind, abandoning the common use of an event frame and instead operating using only a small history of relevant events, allowing latency to scale independently of the sensor resolution. This change in processing paradigm improved the estimation of flow directions by up to 73% compared to the existing method and yielded a demonstrated hARMS throughput of up to 1.21 Mevent/s on the benchmark configuration selected. This throughput enables real-time performance and makes it the fastest known realization of aperture-robust, event-based optical flow to date.INDEX TERMS Event-based, aperture robust, optical flow, neuromorphic computing, field programmable gate arrays, system-on-chip, parallel acceleration, real-time systems

show abstract

Spike-FlowNet: Event-Based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks

Lee

Kosta

Zhu

et al. 2020

Lecture Notes in Computer Science

118

123

View full text Add to dashboard Cite

Event-based Plane-fitting Optical Flow for Dynamic Vision Sensors in FPGA

Cited by 20 publications

References 19 publications

Unsupervised Learning of a Hierarchical Spiking Neural Network for Optical Flow Estimation: From Events to Global Motion Perception

Unsupervised Learning of a Hierarchical Spiking Neural Network for Optical Flow Estimation: From Events to Global Motion Perception

hARMS: A Hardware Acceleration Architecture for Real-Time Event-Based Optical Flow

Spike-FlowNet: Event-Based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks

Contact Info

Product

Resources

About