Exploiting Event Cameras for Spatio-Temporal Prediction of Fast-Changing Trajectories

Monforte, Marco; Arriandiaga, Ander; Glover, Arren; Bartolozzi, Chiara

doi:10.1109/aicas48895.2020.9073855

Cited by 6 publications

(5 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our solution can also be compared with other neuromorphic systems for tracking objects by using artificial neural networks. For example, Monforte et al [24] have developed a recurrent neural network (RNN) for predicting the trajectory of a bouncing ball (the hardware used was ATIS event camera and iCub robot [16,21,24]). The input events of this neural network were also pre-processed by sub-sampling.…”

Section: Discussionmentioning

confidence: 99%

“…This strategy reduces the computational complexity, but can affect the resolution precision. The RNN was using the long-short term memory (LSTM) learning rule [24], which indicates the current output is depended on previous input events. It is similar to our algorithm where the current prediction of the position of ball's arrival is based on 20 latest spikes received from SpiNNaker (Figure 4).…”

Section: Discussionmentioning

confidence: 99%

“…In order to evaluate the performance of its closed-loop execution and also demonstrate the benefit of using SNN in such applications, the platform is implemented on a robot goalkeeper. A similar problem of trajectory prediction for robotics, which utilises an event-based camera and a neural network trained to predict the ball's position has been also investigated by Monforte et al [24].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Neuromorphic Robotic Platform with Visual Input, Processor and Actuator, Based on Spiking Neural Networks

Cheng

Mirza

Nikolić

2020

ASI

View full text Add to dashboard Cite

This paper describes the design and modus of operation of a neuromorphic robotic platform based on SpiNNaker, and its implementation on the goalkeeper task. The robotic system utilises an address event representation (AER) type of camera (dynamic vision sensor (DVS)) to capture features of a moving ball, and a servo motor to position the goalkeeper to intercept the incoming ball. At the backbone of the system is a microcontroller (Arduino Due) which facilitates communication and control between different robot parts. A spiking neuronal network (SNN), which is running on SpiNNaker, predicts the location of arrival of the moving ball and decides where to place the goalkeeper. In our setup, the maximum data transmission speed of the closed-loop system is approximately 3000 packets per second for both uplink and downlink, and the robot can intercept balls whose speed is up to 1 m/s starting from the distance of about 0.8 m. The interception accuracy is up to 85%, the response latency is 6.5 ms and the maximum power consumption is 7.15 W. This is better than previous implementations based on PC. Here, a simplified version of an SNN has been developed for the ‘interception of a moving object’ task, for the purpose of demonstrating the platform, however a generalised SNN for this problem is a nontrivial problem. A demo video of the robot goalie is available on YouTube.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Neuromorphic Robotic Platform with Visual Input, Processor and Actuator, Based on Spiking Neural Networks

Cheng

Mirza

Nikolić

2020

ASI

View full text Add to dashboard Cite

show abstract

“…Tracking can be done by contrast maximization [12], [13] or a globally optimal search [14]. [15] showcases the advantages of using event cameras compared to traditional cameras to track bouncing balls using long-short-term memory (LSTM) architectures. Related to our task, [16] detects ball positions by applying a Hough transform to identify full circles projected onto the image frame.…”

Section: B Event Camerasmentioning

confidence: 99%

EV-Catcher: High-Speed Object Catching Using Low-Latency Event-Based Neural Networks

Wang

Cladera

Bisulco

et al. 2022

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

Event-based sensors have recently drawn increasing interest in robotic perception due to their lower latency, higher dynamic range, and lower bandwidth requirements compared to standard CMOS-based imagers. These properties make them ideal tools for real-time perception tasks in highly dynamic environments. In this work, we demonstrate an application where event cameras excel: accurately estimating the impact location of fast-moving objects. We introduce a lightweight event representation called Binary Event History Image (BEHI) to encode event data at low latency, as well as a learning-based approach that allows real-time inference of a confidence-enabled control signal to the robot. To validate our approach, we present an experimental catching system in which we catch fast-flying pingpong balls. We show that the system is capable of achieving a success rate of 81% in catching balls targeted at different locations, with a velocity of up to 13 m/s even on computeconstrained embedded platforms such as the Nvidia Jetson NX.

show abstract

“…These advantageous properties make event-based processing particularly well-suited for robotics applications. Most of the latest saliency-based approaches use Convolutional Neural Networks (CNNs) [9,10] which do not exploit the event-based sparse computation advantageous for robotic applications [11,12], especially when fully spiking-based methods are employed [13,14].…”

Section: Introductionmentioning

confidence: 99%

Event-Driven Figure-Ground Organisation model for the humanoid robot iCub

D'Angelo,

Voto,

Iacono

et al. 2024

Preprint

Self Cite

View full text Add to dashboard Cite

Figure-ground organisation is a mechanism of perceptual grouping that serves the detection of objects and their boundaries, needed by any agent to interact with the environment. Current approaches to figure-ground segmentation rely on classical computer vision or deep learning, requiring vast computational resources, in particular during the training phase of the latter. Inspired by the primate visual system, we developed a bio-inspired perception system to compute figure-ground organisation for the neuromorphic robot iCub. The proposed model distinguishes foreground objects from the background, thanks to a hierarchical biologically plausible architecture, exploiting event-driven vision. Differing from classical approaches, the use of event-driven cameras reduces data redundancy and computation. The system has been qualitatively and quantitatively assessed both in simulation and using the event-driven cameras mounted on the iCub looking at different scenarios and stimuli. The model successfully segments items in diverse real-world scenarios, showing comparable results against its frame-based version on simple stimuli and on the Berkeley Segmentation dataset benchmark (D. Martin, 2001). This model further offers a valuable addition to hybrid systems, complementing conventional computer vision deep learning models by processing only relevant data in detected Regions of Interest (ROI), thereby enabling low-latency autonomous robotic applications without redundant information.

show abstract

Exploiting Event Cameras for Spatio-Temporal Prediction of Fast-Changing Trajectories

Cited by 6 publications

References 34 publications

Neuromorphic Robotic Platform with Visual Input, Processor and Actuator, Based on Spiking Neural Networks

Neuromorphic Robotic Platform with Visual Input, Processor and Actuator, Based on Spiking Neural Networks

EV-Catcher: High-Speed Object Catching Using Low-Latency Event-Based Neural Networks

Event-Driven Figure-Ground Organisation model for the humanoid robot iCub

Contact Info

Product

Resources

About