Proto-object based saliency for event-driven cameras

Iacono, Massimiliano; D’Angelo, Giulia; Glover, Arren; Tikhanoff, Vadim; Niebur, Ernst; Bartolozzi, Chiara

doi:10.1109/iros40897.2019.8967943

Cited by 9 publications

(13 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Working towards the implementation of robots with full neuromorphic vision, the neuromorphic and computational neuroscience communities have started in-depth work on perceptive modules for stereo vision 34 and vergence 35 , attention 36 , and object recognition 37 . These algorithms can run on neuromorphic computing substrates for exploiting efficiency, adaptability and low latency.…”

Section: Box 1 | the Need For Adaptation In Roboticsmentioning

confidence: 99%

Embodied neuromorphic intelligence

2022

Self Cite

View full text Add to dashboard Cite

The design of robots that interact autonomously with the environment and exhibit complex behaviours is an open challenge that can benefit from understanding what makes living beings fit to act in the world. Neuromorphic engineering studies neural computational principles to develop technologies that can provide a computing substrate for building compact and low-power processing systems. We discuss why endowing robots with neuromorphic technologies – from perception to motor control – represents a promising approach for the creation of robots which can seamlessly integrate in society. We present initial attempts in this direction, highlight open challenges, and propose actions required to overcome current limitations.

show abstract

Section: Box 1 | the Need For Adaptation In Roboticsmentioning

confidence: 99%

Embodied neuromorphic intelligence

2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…This work takes inspiration from the bio-inspired saliency-based proto-object model for frame-based cameras initially proposed by Russell et al [21] and its event-camera adaptation [31]. The former is composed of three channels: intensity, colour opponency and orientation, competing with each other to represent the scene.…”

Section: Event-based Spiking Neural Network Proto-object Saliency Modelmentioning

confidence: 99%

“…The two processes are different but the related outcome, the edge extraction and the contrast information, are similar. These inherent capabilities can be used as substitutes for the first two layers of processing in the event-based version of the saliency-based model [31]: CS filtering and edge extraction. In fact, assuming a dynamic scene where a dark object is moving over a white background the leading edge would produce negative events and the trailing edge positive events, therefore providing information about the object contrast with respect to the background.…”

Section: Event-based Spiking Neural Network Proto-object Saliency Modelmentioning

confidence: 99%

“…Rea et al [28] exploited visual attention for a bio-inspired pipeline using event-driven cameras (ATIS cameras) [29] mounted on iCub, the neuromorphic robot [30]. This implementation [28] exploits the low latency of the event cameras, further increasing the speed of the response towards online attention, but does not include the proto-object concept, that was later included by modifying a frame-based proto-object model [21] in a way that is suitable for event-based cameras [31]. The implementation proposed by Iacono et al [31] adapts the proto-object model based on RGB cameras to event-driven input, using the contrast feature maps naturally encoded by event-driven cameras.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Event driven bio-inspired attentive system for the iCub humanoid robot on SpiNNaker

D’Angelo

Perrett

Iacono

et al. 2022

Neuromorph. Comput. Eng.

Self Cite

View full text Add to dashboard Cite

Attention leads the gaze of the observer towards interesting items, allowing a detailed analysis only for selected regions of a scene. A robot can take advantage of the perceptual organisation of the features in the scene to guide its attention to better understand its environment. Current bottom–up attention models work with standard RGB cameras requiring a significant amount of time to detect the most salient item in a frame-based fashion. Event-driven cameras are an innovative technology to asynchronously detect contrast changes in the scene with a high temporal resolution and low latency. We propose a new neuromorphic pipeline exploiting the asynchronous output of the event-driven cameras to generate saliency maps of the scene. In an attempt to further decrease the latency, the neuromorphic attention model is implemented in a spiking neural network on SpiNNaker, a dedicated neuromorphic platform. The proposed implementation has been compared with its bio-inspired GPU counterpart, and it has been benchmarked against ground truth fixational maps. The system successfully detects items in the scene, producing saliency maps comparable with the GPU implementation. The asynchronous pipeline achieves an average of 16 ms latency to produce a usable saliency map.

show abstract

“…A proto-object based saliency model was shown to predict eye fixations with good accuracy (Russell et al, 2014). While originally this model used information from maps of intensity, color, and orientation, it was later extended to additionally utilize motion, depth, and texture features (Hu et al, 2016; Mancinelli et al, 2018; Molin et al, 2015; Uejima et al, 2020), and also implemented in biofidelic neuromorphic hardware (Iacono et al, 2019).…”

Section: Introductionmentioning

confidence: 99%

The influence of stereopsis on visual saliency in a proto-object based model of selective attention

Uejima

Mancinelli

Niebur

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Some animals including humans use stereoscopic vision which reconstructs spatial information about the environment from the disparity between images captured by eyes in two separate adjacent locations. Like other sensory information, such stereoscopic information is expected to influence attentional selection. We develop a biologically plausible model of binocular vision to study its effect on bottom-up visual attention, i.e., visual saliency. In our model, the scene is organized in terms of proto-objects on which attention acts, rather than on unbound sets of elementary features. We show that taking into account the stereoscopic information improves the performance of the model in the prediction of human eye movements with statistically significant differences.

show abstract

Proto-object based saliency for event-driven cameras

Cited by 9 publications

References 24 publications

Embodied neuromorphic intelligence

Embodied neuromorphic intelligence

Event driven bio-inspired attentive system for the iCub humanoid robot on SpiNNaker

The influence of stereopsis on visual saliency in a proto-object based model of selective attention

Contact Info

Product

Resources

About