Instantaneous Stereo Depth Estimation of Real-World Stimuli with a Neuromorphic Stereo-Vision Setup

Risi, Nicoletta; Calabrese, Enrico; Indiveri, Giacomo

doi:10.1109/iscas51556.2021.9401402

Cited by 6 publications

(5 citation statements)

References 19 publications

(20 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Historically, several groups used bio-inspired approaches such as Spiking Neural Networks (SNNs), in a very hardware-oriented direction, but not in a "deep learning" setting. For instance, the authors of [18] implemented a spike-based algorithm on a FPGA to regress low-resolution depth maps on a small size dataset. Furthermore, [19] proposed a SNN for processing depth from defocus (DFD); this work targeted neuromorphic chips and was able of recovering depth at full resolution, but reconstructions were not dense and the approach was not based on learning.…”

Section: Related Workmentioning

confidence: 99%

StereoSpike: Depth Learning With a Spiking Neural Network

Rançon¹,

Cuadrado-Anibarro²,

Cottereau³

et al. 2022

IEEE Access

View full text Add to dashboard Cite

Depth estimation is an important computer vision task, useful in particular for navigation in autonomous vehicles, or for object manipulation in robotics. Here, we propose to solve it using StereoSpike, an end-to-end neuromorphic approach, combining two event-based cameras and a Spiking Neural Network (SNN) with a modified U-Net-like encoder-decoder architecture. More specifically, we used the Multi Vehicle Stereo Event Camera Dataset (MVSEC). It provides a depth ground-truth, which was used to train StereoSpike in a supervised manner, using surrogate gradient descent. We propose a novel readout paradigm to obtain a dense analog prediction -the depth of each pixel-from the spikes of the decoder. We demonstrate that this architecture generalizes very well, even better than its non-spiking counterparts, leading to near state-of-the-art test accuracy. To the best of our knowledge, it is the first time that such a large-scale regression problem is solved by a fully spiking neural network. Finally, we show that very low firing rates (<5%) can be obtained via regularization, with a minimal cost in accuracy. This means that StereoSpike could be efficiently implemented on neuromorphic chips, opening the door for low power and real time embedded systems.

show abstract

Section: Related Workmentioning

confidence: 99%

StereoSpike: Depth Learning With a Spiking Neural Network

Rançon¹,

Cuadrado-Anibarro²,

Cottereau³

et al. 2022

IEEE Access

View full text Add to dashboard Cite

show abstract

“…A survey of the developed specialized spike-aware algorithms for image processing and depth estimation by event-based cameras can be found in a recent review paper [85]. The exploitation of event-based image representation is demonstrated in [86], where two streams of event camera data are passed into a SNN architecture composed of two cooperative populations (one for coincidence and one for disparity) to produce instantaneous stereo depth perception with real-world stimuli. The work of [87] similarly uses two spiking neural populations connected to two neuromorphic cameras to solve the stereo correspondence problem.…”

Section: Snn-based Vision For Robotic Perceptionmentioning

confidence: 99%

From Brain Models to Robotic Embodied Cognition: How Does Biological Plausibility Inform Neuromorphic Systems?

Pham,

D’Angiulli,

Dehnavi

et al. 2023

Brain Sciences

View full text Add to dashboard Cite

We examine the challenging “marriage” between computational efficiency and biological plausibility—A crucial node in the domain of spiking neural networks at the intersection of neuroscience, artificial intelligence, and robotics. Through a transdisciplinary review, we retrace the historical and most recent constraining influences that these parallel fields have exerted on descriptive analysis of the brain, construction of predictive brain models, and ultimately, the embodiment of neural networks in an enacted robotic agent. We study models of Spiking Neural Networks (SNN) as the central means enabling autonomous and intelligent behaviors in biological systems. We then provide a critical comparison of the available hardware and software to emulate SNNs for investigating biological entities and their application on artificial systems. Neuromorphics is identified as a promising tool to embody SNNs in real physical systems and different neuromorphic chips are compared. The concepts required for describing SNNs are dissected and contextualized in the new no man’s land between cognitive neuroscience and artificial intelligence. Although there are recent reviews on the application of neuromorphic computing in various modules of the guidance, navigation, and control of robotic systems, the focus of this paper is more on closing the cognition loop in SNN-embodied robotics. We argue that biologically viable spiking neuronal models used for electroencephalogram signals are excellent candidates for furthering our knowledge of the explainability of SNNs. We complete our survey by reviewing different robotic modules that can benefit from neuromorphic hardware, e.g., perception (with a focus on vision), localization, and cognition. We conclude that the tradeoff between symbolic computational power and biological plausibility of hardware can be best addressed by neuromorphics, whose presence in neurorobotics provides an accountable empirical testbench for investigating synthetic and natural embodied cognition. We argue this is where both theoretical and empirical future work should converge in multidisciplinary efforts involving neuroscience, artificial intelligence, and robotics.

show abstract

“…The +1λ stimulus shown in figure 8a thus activates . They may also postulate inhibitory connections between disparity sensors corresponding to the same location in one eye (oblique blue lines, as in [72]), and/ or between disparity sensors corresponding to the same visual direction (vertical blue lines, as in [71]). 8b).…”

Section: (F ) Additional Processing Required For Correspondencementioning

confidence: 99%

“…Stereo algorithms often implement mutual excitation between disparity sensors tuned to similar disparities at different cyclopean locations (shown here by gold horizontal lines). They may also postulate inhibitory connections between disparity sensors corresponding to the same location in one eye (oblique blue lines, as in [ 72 ]), and/or between disparity sensors corresponding to the same visual direction (vertical blue lines, as in [ 71 ]). …”

Section: Stereopsis and Correspondencementioning

confidence: 99%

“…This builds in a preference for smoothness (gradual change in disparity with azimuth) and uniqueness (each retinal location is assigned only one disparity). The influential early stereo algorithm PMF [ 70 ] used a disparity gradient limit to achieve a similar result, while a recent stereo vision system using event-based sensing and spike-based neuromorphic hardware [ 71 ] has each disparity sensor receiving feed-forward excitation from a ‘coincidence neuron’ tuned to the sensor disparity, but feed-forward inhibitory input from all other coincidence neurons tuned to the same cyclopean position (vertical blue lines in figure 4 b , c ). In machine stereo algorithms, this is usually achieved in an energy-minimization framework [ 33 , 51 ].…”

Section: Stereopsis and Correspondencementioning

confidence: 99%

See 1 more Smart Citation

Stereopsis without correspondence

Read

2022

Phil. Trans. R. Soc. B

View full text Add to dashboard Cite

Stereopsis has traditionally been considered a complex visual ability, restricted to large-brained animals. The discovery in the 1980s that insects, too, have stereopsis, therefore, challenged theories of stereopsis. How can such simple brains see in three dimensions? A likely answer is that insect stereopsis has evolved to produce simple behaviour, such as orienting towards the closer of two objects or triggering a strike when prey comes within range. Scientific thinking about stereopsis has been unduly anthropomorphic, for example assuming that stereopsis must require binocular fusion or a solution of the stereo correspondence problem. In fact, useful behaviour can be produced with very basic stereoscopic algorithms which make no attempt to achieve fusion or correspondence, or to produce even a coarse map of depth across the visual field. This may explain why some aspects of insect stereopsis seem poorly designed from an engineering point of view: for example, paying no attention to whether interocular contrast or velocities match. Such algorithms demonstrably work well enough in practice for their species, and may prove useful in particular autonomous applications. This article is part of a discussion meeting issue ‘New approaches to 3D vision’.

show abstract

Instantaneous Stereo Depth Estimation of Real-World Stimuli with a Neuromorphic Stereo-Vision Setup

Cited by 6 publications

References 19 publications

StereoSpike: Depth Learning With a Spiking Neural Network

StereoSpike: Depth Learning With a Spiking Neural Network

From Brain Models to Robotic Embodied Cognition: How Does Biological Plausibility Inform Neuromorphic Systems?

Stereopsis without correspondence

Contact Info

Product

Resources

About