Event driven bio-inspired attentive system for the iCub humanoid robot on SpiNNaker

Abstract

Attention leads the gaze of the observer towards interesting items, allowing a detailed analysis only for selected regions of a scene. A robot can take advantage of the perceptual organisation of the features in the scene to guide its attention to better understand its environment. Current bottom–up attention models work with standard RGB cameras requiring a significant amount of time to detect the most salient item in a frame-based fashion. Event-driven cameras are an innovative technology to asynchronously detect contrast changes in the scene with a high temporal resolution and low latency. We propose a new neuromorphic pipeline exploiting the asynchronous output of the event-driven cameras to generate saliency maps of the scene. In an attempt to further decrease the latency, the neuromorphic attention model is implemented in a spiking neural network on SpiNNaker, a dedicated neuromorphic platform. The proposed implementation has been compared with its bio-inspired GPU counterpart, and it has been benchmarked against ground truth fixational maps. The system successfully detects items in the scene, producing saliency maps comparable with the GPU implementation. The asynchronous pipeline achieves an average of 16 ms latency to produce a usable saliency map.

Event-Driven Figure-Ground Organisation model for the humanoid robot iCub

“…All of these implementations brought to the design of the figure-ground organisation model detecting protoobjects and figure-ground assignment exploiting the feedback and feedforward connections and filtering with the Von Mises kernel [21]. The proposed model takes inspiration from [21] and it is a follow-up of previous event-driven attempts to detect proto-objects [22,48,49]. The first event-based implementation of [25] has been proposed by Iacono et.…”

Simultaneous neuromorphic selection of multiple salient objects for event vision

“…Table II compares our contribution with the neuromorphic saliency models implemented in [17], [19] as well as with our initial model [5]. The data presented here were either retrieved directly from the information given by the authors or calculated from the description of each model.…”

“…TABLE II: Comparison between our contribution and the state-of-the-art, with input data of size w × h (w the width and h the height), OL the overlapping percentage described in [17] and div the dividing factor between the input layer and the saliency detector in [5]. A numerical value was calculated for each theoretical estimation, for w = h = 128, OL = 5% and div = 16.…”

“…However, works that address all of these constraints are rare in the literature (see [11]): for example, the models introduced in [12]- [14] detect saliency on event data with traditional neural networks while [15], [16] implement neuromorphic models of visual saliency in RGB data. Additionally, neuromorphic saliency detection models applied to event data are most often than not derived from existing models implemented with traditional neural networks and applied to RGB or grayscale images: [17] is for example adapted from [13], itself originally adapted from the grouping mechanism estimating the location and spatial scale of protoobjects in RGB data implemented in [18]. Only a few models truly take advantage of the intrinsic dynamics of SNNs and the uniqueness of event data: in particular, [19] make use of the mathematical model of Dynamic Neural Field [20] as a soft Winner-Takes-All (WTA) to implement salient tracking of pre-activated objects.…”

Closing the loop: High-speed robotics with accelerated neuromorphic hardware

“…This poses strong timing requirements on the—simulated—nervous system, which needs to interact with its surroundings in realtime. Especially for biologically inspired spiking neural networks (SNNs), this prerequisite has led to the deployment of specialized neuromorphic accelerators for neurorobotic tasks (Richter et al, 2016 ; Blum et al, 2017 ; Milde et al, 2017 ; Kreiser et al, 2018 ; Yan et al, 2021 ; DAngelo et al, 2022 ). These systems integrate specific analog or digital modules for the efficient simulation—or emulation —of spiking neurons and are thereby capable of running such networks at biological realtime.…”