Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccades

Abstract

Creating datasets for Neuromorphic Vision is a challenging task. A lack of available recordings from Neuromorphic Vision sensors means that data must typically be recorded specifically for dataset creation rather than collecting and labeling existing data. The task is further complicated by a desire to simultaneously provide traditional frame-based recordings to allow for direct comparison with traditional Computer Vision algorithms. Here we propose a method for converting existing Computer Vision static image datasets into Neuromorphic Vision datasets using an actuated pan-tilt camera platform. Moving the sensor rather than the scene or image is a more biologically realistic approach to sensing and eliminates timing artifacts introduced by monitor updates when simulating motion on a computer monitor. We present conversion of two popular image datasets (MNIST and Caltech101) which have played important roles in the development of Computer Vision, and we provide performance metrics on these datasets using spike-based recognition algorithms. This work contributes datasets for future use in the field, as well as results from spike-based algorithms against which future works can compare. Furthermore, by converting datasets already popular in Computer Vision, we enable more direct comparison with frame-based approaches.

Invariant feature extraction from event based stimuli

“…The "Event based MNIST" database [20] proved to be the most difficult task because of the variability of the original hand written images [21]. The data set contains 60,000 training instances and 10,000 testing instances.…”

“…Letters and Digits 100% 100% [11] Faces 100% 79% [11] Event based MNIST 89.9% 83.4% [20] VI. DISCUSSION AND CONCLUSION…”

Effective sensor fusion with event-based sensors and deep network architectures

“…The approach taken in collecting the N-MNIST dataset [13] was to present each digit to a DVS while moving the image sensor in a controlled way. The N-MNIST dataset includes a script to counter the movement of the image sensor so that the triggered events can also be centered in the original image position.…”

“…The network accuracies show that processing the cochlear spikes is more challenging than using the MFCCs of the audio, but overall classification accuracy is still quite high. Indeed the recognition accuracy of many of these networks (Intensity, Canny, N-MNIST, and Cochleagram CNN) establishes a new state-of-the-art benchmark ( [9], [13]). …”

“…The performance of these networks is tested on two standard benchmark datasets: the MNIST handwritten digit recognition dataset and the TIDIGITS audio dataset. In addition, the corresponding spike databases consisting of recordings through the DVS on MNIST (N-MNIST) [13], and recordings through the DAS cochlea on TIDIGITS [14] are used in the evaluation of the networks. Section II describes the data processing methods for the eventbased sensors as well as the deep network architectures used.…”