Event camera object recognition using spatiotemporal event time surface and reward-modulated spike-timing-dependent plasticity learning rule

Abstract

Event cameras record moving objects with asynchronous event streams. It remains a challenge to make full use of the spatiotemporal information of event streams to extract high-quality features and to make event camera object recognition. We propose an event-based event camera object recognition system, which includes a denoising module and an object recognition module. The denoising module removes noise events using spatiotemporal information of the events. The object recognition module extracts primary spatiotemporal features based on the event time surface prototypes obtained by clustering, then further extracts complex diagnostic features, and makes object recognition using spiking neural networks with reward-modulated spike-timing-dependent plasticity learning rule. Experimental results demonstrate that our system has better performance than baseline methods on five popular event camera datasets, especially on datasets with rich spatiotemporal dynamics. Our denoising method improves the noise robustness of our event camera object recognition system on high-speed moving datasets that are greatly affected by noises. Moreover, our method has much better recognition ability than baseline methods when using short input event streams. Our method is very beneficial for developing event-based event camera object recognition algorithm when event streams are short or noises are serious.