A Low Power, Fully Event-Based Gesture Recognition System

Amir, Arnon; Taba, Brian; Berg, David J. Van Den; Melano, Timothy; McKinstry, Jeffrey L.; Nolfo, Carmelo di; Nayak, T. K.; Andreopoulos, Alexander; Garreau, Guillaume J.; Mendoza, Marcela; Kusnitz, Jeff; DeBole, Michael; Esser, Steven K.; Delbrück, Tobi; Flickner, Myron; Modha, Dharmendra S.

doi:10.1109/cvpr.2017.781

Cited by 531 publications

(471 citation statements)

References 41 publications

Supporting

Mentioning

469

Contrasting

Order By: Relevance

“…Duration(0.25s) Duration(0.5s) LSTM [54] 0.882 0.865 PointNet [50] 0.887 0.902 PointNet++ [51] 0.923 0.941 Amir CVPR2017 [2] -0.945 Wang WACV2019 [64] 0.940 0.953 ResNet 34 [24] 0.943 0.955 I3D [10] 0.951 0.965 RG-CNN + Plain 3D 0.954 0.968 RG-CNN + Incep. 3D 0.957 0.968 RG-CNN + Res.…”

Section: Methodsmentioning

confidence: 99%

“…Both datasets were collected from an experimental setting environment with a monotonous background, and relative to equivalent datasets for APS-based evaluation datasets, both are modest in their size and class count; as such, they cannot represent complex real-life scenarios and are not robust to evaluation for advanced algorithms. Moreover, previous work [2,66,46,64] already achieves high accuracies on them. This is why, it is necessary to establish larger and more complex datasets for the evaluation of our proposal and for future proposals on NVS-based action recognition.…”

Section: Action Recognitionmentioning

confidence: 96%

See 1 more Smart Citation

Graph-Based Object Classification for Neuromorphic Vision Sensing

Chadha

Abbas

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

113

View full text Add to dashboard Cite

Neuromorphic vision sensing (NVS) devices represent visual information as sequences of asynchronous discrete events (a.k.a., "spikes") in response to changes in scene reflectance. Unlike conventional active pixel sensing (APS), NVS allows for significantly higher event sampling rates at substantially increased energy efficiency and robustness to illumination changes. However, feature representation for NVS is far behind its APS-based counterparts, resulting in lower performance in high-level computer vision tasks. To fully utilize its sparse and asynchronous nature, we propose a compact graph representation for NVS, which allows for end-to-end learning with graph convolution neural networks. We couple this with a novel end-to-end feature learning framework that accommodates both appearancebased and motion-based tasks. The core of our framework comprises a spatial feature learning module, which utilizes residual-graph convolutional neural networks (RG-CNN), for end-to-end learning of appearance-based features directly from graphs. We extend this with our proposed Graph2Grid block and temporal feature learning module for efficiently modelling temporal dependencies over multiple graphs and a long temporal extent. We show how our framework can be configured for object classification, action recognition and action similarity labeling. Importantly, our approach preserves the spatial and temporal coherence of spike events, while requiring less computation and memory. The experimental validation shows that our proposed framework outperforms all recent methods on standard datasets. Finally, to address the absence of large realworld NVS datasets for complex recognition tasks, we introduce, evaluate and make available the American Sign Language letters (ASL-DVS), as well as human action dataset (UCF101-DVS, HMDB51-DVS and ASLAN-DVS). Figure 1: Examples of archery action captured by APS and NVS sensors. APS sensors capture images at fixed frame rates, while NVS sensors output a stream of events.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Action Recognitionmentioning

confidence: 96%

Graph-Based Object Classification for Neuromorphic Vision Sensing

Chadha

Abbas

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

113

View full text Add to dashboard Cite

show abstract

“…Object detection tasks require a clear representation of the object boundaries that define the shape of the object-of-interest. Recall (2). To successfully filter events prior to time-surface generation, we propose the following:…”

Section: Proposed Methods: Inceptive Event Time-surfacesmentioning

confidence: 99%

“…arXiv:2002.11656v1 [cs.NE] 26 Feb 2020 these neuromorphically inspired cameras can operate at extremely high temporal resolution (>800kHz), low latency (20 microseconds), wide dynamic range (> 120dB), and low power (30mW). They report only changes in the pixel intensity, requiring a new set of techniques to perform basic image processing and computer vision tasks-examples include optical flow [3,8], feature extraction [4,12,13], gesture recognition [2,11], and object recognition [5,14].…”

Section: Introductionmentioning

confidence: 99%

Inceptive Event Time-Surfaces for Object Classification Using Neuromorphic Cameras

Baldwin

Almatrafi

Kaufman

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

This paper presents a novel fusion of low-level approaches for dimensionality reduction into an effective approach for high-level objects in neuromorphic camera data called Inceptive Event Time-Surfaces (IETS). IETSs overcome several limitations of conventional time-surfaces by increasing robustness to noise, promoting spatial consistency, and improving the temporal localization of (moving) edges. Combining IETS with transfer learning improves state-of-the-art performance on the challenging problem of object classification utilizing event camera data.

show abstract

“…The IBM DvsGesture [17] dataset consists of recordings of 29 different individuals performing 10 different actions such as clapping and an unspecified gesture for a total of 11 classes. The actions are recorded using a DVS camera, an event-based neuromorphic sensor, under three different lighting conditions.…”

Section: Experimental Evaluation a Setupmentioning

confidence: 99%

On-chip Few-shot Learning with Surrogate Gradient Descent on a Neuromorphic Processor

Stewart

Orchard

Shrestha

et al. 2020

2020 2nd IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS)

View full text Add to dashboard Cite

Recent work suggests that synaptic plasticity dynamics in biological models of neurons and neuromorphic hardware are compatible with gradient-based learning [1]. Gradientbased learning requires iterating several times over a dataset, which is both time-consuming and constrains the training samples to be independently and identically distributed. This is incompatible with learning systems that do not have boundaries between training and inference, such as in neuromorphic hardware. One approach to overcome these constraints is transfer learning, where a portion of the network is pre-trained and mapped into hardware and the remaining portion is trained online. Transfer learning has the advantage that pre-training can be accelerated offline if the task domain is known, and few samples of each class are sufficient for learning the target task at reasonable accuracies. Here, we demonstrate on-line surrogate gradient few-shot learning on Intel's Loihi neuromorphic research processor using features pre-trained with spikebased gradient backpropagation-through-time. Our experimental results show that the Loihi chip can learn gestures online using a small number of shots and achieve results that are comparable to the models simulated on a conventional processor.

show abstract

A Low Power, Fully Event-Based Gesture Recognition System

Cited by 531 publications

References 41 publications

Graph-Based Object Classification for Neuromorphic Vision Sensing

Graph-Based Object Classification for Neuromorphic Vision Sensing

Inceptive Event Time-Surfaces for Object Classification Using Neuromorphic Cameras

On-chip Few-shot Learning with Surrogate Gradient Descent on a Neuromorphic Processor

Contact Info

Product

Resources

About