A spiking neural network model of 3D perception for event-based neuromorphic stereo vision systems

Abstract

Stereo vision is an important feature that enables machine vision systems to perceive their environment in 3D. While machine vision has spawned a variety of software algorithms to solve the stereo-correspondence problem, their implementation and integration in small, fast, and efficient hardware vision systems remains a difficult challenge. Recent advances made in neuromorphic engineering offer a possible solution to this problem, with the use of a new class of event-based vision sensors and neural processing devices inspired by the organizing principles of the brain. Here we propose a radically novel model that solves the stereo-correspondence problem with a spiking neural network that can be directly implemented with massively parallel, compact, low-latency and low-power neuromorphic engineering devices. We validate the model with experimental results, highlighting features that are in agreement with both computational neuroscience stereo vision theories and experimental findings. We demonstrate its features with a prototype neuromorphic hardware system and provide testable predictions on the role of spike-based representations and temporal dynamics in biological stereo vision processing systems.

Deep Learning With Spiking Neurons: Opportunities and Challenges

“…We are currently observing the interesting trend that event-based vision becomes increasingly interesting for research communities rooted in classical computer vision and robotics. Advantages of using event-based sensors have been demonstrated for diverse applications such as tracking (Mueggler et al, 2014 ; Lagorce et al, 2015 ; Gallego et al, 2018 ), stereo vision (Rogister et al, 2012 ; Osswald et al, 2017 ; Martel et al, 2018 ), optical flow estimation (Benosman et al, 2014 ; Bardow et al, 2016 ), gesture recognition (Lee et al, 2014 ; Amir et al, 2017 ), scene reconstruction (Carneiro et al, 2013 ; Kim et al, 2016 ; Rebecq et al, 2017 ), or SLAM (Weikersdorfer et al, 2014 ; Vidal et al, 2018 ). All of these applications benefit from the high speed and the high dynamic range of spike-based sensors to solve tasks, such as high-speed localization and navigation, which are very hard with conventional vision sensors.…”

“…In addition, using time domain input is additional valuable information compared to frame-driven approaches, where an artificial time step imposed by the sensor is introduced. This can lead to efficient computation of features such as optical flow (Benosman et al, 2014 ) or stereo disparity (Osswald et al, 2017 ), and in combination with learning rules sensitive to spike timing leads to more data-efficient training (Panda et al, 2017 ).…”

Focus Is All You Need: Loss Functions for Event-Based Vision

“…In this work, we build upon the motion compensation framework [35] and extend it to include twenty more loss functions for applications such as ego-motion, depth and Variance (4) [33,35] Statistical No max Mean Square (9) [33,36] Statistical No max Mean Absolute Deviation (10) Statistical No max Mean Absolute Value (11) Statistical No max Entropy (12) Statistical No max Image Area (8) Statistical No min Image Range (13) Statistical No max Local Variance (14) Statistical [37] Statistical No min Table 1: List of objective functions considered.…”

“…Loss Function: Local Variance. The derivative of the aggregated local variance of the IWE is, by the chain rule on(14),…”

Event-Based Attention and Tracking on Neuromorphic Hardware

“…Such event-based sensing allows us to perform some vision tasks extremely efficiently, reducing the amount of required computation, transmitted data, and power consumption. Many eventbased vision pipelines and architectures have been developed over the last decade [1,2], which address such vision tasks as stereo vision [3], 3D pose estimation [4,5], or optical flow [6]. These event-based pipelines are typically implemented on conventional Von Neumann computer architectures.…”