Neuromorphic Vision Sensing for CNN-based Action Recognition

Chadha, Aaron; Bi, Yin; Abbas, Abdulrahman; Andreopoulos, Yiannis

doi:10.1109/icassp.2019.8683606

Cited by 17 publications

(15 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similary, Ghosh et al partitioned events into a three-dimensional grid of voxels where spatio-temporal filters are used to learn the features, and learnt features are fed as input to CNNs for action recognition [16]. Chadha et al [31] generated frames by summing the polarity of events in each address as pixel, then fed them into a multi-modal teacher-student framework for action recognition. While useful for early-stage attempts, these frame-based methods are not well-suited for the neuromorphic event's sparse and asynchronous nature since the frame sizes that need to be processed are substantially larger than those of the original NVS streams.…”

Section: Related Workmentioning

confidence: 99%

Graph-Based Spatio-Temporal Feature Learning for Neuromorphic Vision Sensing

Bi¹,

Chadha²,

Abbas³

et al. 2020

IEEE Trans. on Image Process.

View full text Add to dashboard Cite

Neuromorphic vision sensing (NVS) devices represent visual information as sequences of asynchronous discrete events (a.k.a., "spikes") in response to changes in scene reflectance. Unlike conventional active pixel sensing (APS), NVS allows for significantly higher event sampling rates at substantially increased energy efficiency and robustness to illumination changes. However, feature representation for NVS is far behind its APS-based counterparts, resulting in lower performance in high-level computer vision tasks. To fully utilize its sparse and asynchronous nature, we propose a compact graph representation for NVS, which allows for end-to-end learning with graph convolution neural networks. We couple this with a novel end-to-end feature learning framework that accommodates both appearancebased and motion-based tasks. The core of our framework comprises a spatial feature learning module, which utilizes residual-graph convolutional neural networks (RG-CNN), for end-to-end learning of appearance-based features directly from graphs. We extend this with our proposed Graph2Grid block and temporal feature learning module for efficiently modelling temporal dependencies over multiple graphs and a long temporal extent. We show how our framework can be configured for object classification, action recognition and action similarity labeling. Importantly, our approach preserves the spatial and temporal coherence of spike events, while requiring less computation and memory. The experimental validation shows that our proposed framework outperforms all recent methods on standard datasets. Finally, to address the absence of large real-world NVS datasets for complex recognition tasks, we introduce, evaluate and make available the American Sign Language letters (ASL-DVS), as well as human action dataset (UCF101-DVS, HMDB51-DVS and ASLAN-DVS). Index Terms-Neuromorphic vision sensing, spatio-temporal feature learning, graph convolutional neural networks, object classification, human action recognition I. INTRODUCTION With the prevalence and advances of CMOS active pixel sensing (APS) and deep learning, researchers have achieved good performance in APS-based computer vision tasks, such as object detection [1], [2], object recognition [3], [4] and action recognition [5], [6]. However, APS cameras suffer from limited frame rate, high redundancy between frames, blurriness YB, AC, AA and YA are with the Electronic and Electrical Engineer

show abstract

Section: Related Workmentioning

confidence: 99%

Graph-Based Spatio-Temporal Feature Learning for Neuromorphic Vision Sensing

Bi¹,

Chadha²,

Abbas³

et al. 2020

IEEE Trans. on Image Process.

View full text Add to dashboard Cite

show abstract

“…They demonstrate the usability of event data in HAR compared to conventional camerabased vision systems, where complex optic flow estimation is required. A similar approach was followed in [40] by using two CNNs to learn features from event frames and corresponding optic flow from the original RGB. Converting the events into frames is also applied in [43] to classify the actions of the neuromorphic version of UCF11 dataset.…”

Section: Related Workmentioning

confidence: 99%

“…The existing work that uses hand-crafted features has achieved an accuracy rate of 75.13% [39], while the works that have used deep learning have achieved accuracy rates ranging from 51.5% to 92.9% [40]- [43] as detailed in Section IV-B. It can be also observed that the accuracy rates of these methods depend on the quality of the constructed event frames.…”

Section: Related Workmentioning

confidence: 99%

Making Sense of Neuromorphic Event Data for Human Action Recognition

2021

View full text Add to dashboard Cite

Neuromorphic vision sensors provide low power sensing and capture salient spatial-temporal events. The majority of the existing neuromorphic sensing work focus on object detection. However, since they only record the events, they provide an efficient signal domain for privacy aware surveillance tasks. This paper explores how the neuromorphic vision sensor data streams can be analysed for human action recognition, which is a challenging application. The proposed method is based on handcrafted features. It consists of a pre-processing step for removing the noisy events followed by the extraction of handcrafted local and global feature vectors corresponding to the underlying human action. The local features are extracted considering a set of high-order descriptive statistics from the spatio-temporal events in a time window slice, while the global features are extracted by considering the frequencies of occurrences of the temporal event sequences. Then, low complexity classifiers, such as, support vector machines (SVM) and K-Nearest Neighbours (KNN), are trained using these feature vectors. The proposed method evaluation uses three groups of datasets: Emulator-based, re-recording-based and native NVS-based. The proposed method has outperformed the existing methods in terms of human action recognition accuracy rates by 0.54%, 19.3% and 25.61% for E-KTH, E-UCF11 and E-HMDB51 datasets, respectively. This paper also reports results for three further datasets: E-UCF50, R-UCF50, and N-Actions, which are reported for the first time for human action recognition on neuromorphic vision sensor domain.INDEX TERMS Neuromorphic vision sensing (NVS), event cameras, dynamic vision sensing (DVS), Human action recognition (HAR), Local features, Global features,

show abstract

“…As to the number of parameters, for each convolution layer in both CNNs and GCNs, it is (C in K 2 + 1)C out , while in fully connected layers, it is (C in + 1)C out . As shown by (11), FLOPs of graph convolution depend on the number of edges and nodes. Since the size of input graph varies per dataset, we opt to report representative results from N-Caltech101 in Table 2.…”

Section: Object Classificationmentioning

confidence: 99%

Graph-Based Object Classification for Neuromorphic Vision Sensing

Chadha

Abbas

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

113

View full text Add to dashboard Cite

Neuromorphic vision sensing (NVS) devices represent visual information as sequences of asynchronous discrete events (a.k.a., "spikes") in response to changes in scene reflectance. Unlike conventional active pixel sensing (APS), NVS allows for significantly higher event sampling rates at substantially increased energy efficiency and robustness to illumination changes. However, feature representation for NVS is far behind its APS-based counterparts, resulting in lower performance in high-level computer vision tasks. To fully utilize its sparse and asynchronous nature, we propose a compact graph representation for NVS, which allows for end-to-end learning with graph convolution neural networks. We couple this with a novel end-to-end feature learning framework that accommodates both appearancebased and motion-based tasks. The core of our framework comprises a spatial feature learning module, which utilizes residual-graph convolutional neural networks (RG-CNN), for end-to-end learning of appearance-based features directly from graphs. We extend this with our proposed Graph2Grid block and temporal feature learning module for efficiently modelling temporal dependencies over multiple graphs and a long temporal extent. We show how our framework can be configured for object classification, action recognition and action similarity labeling. Importantly, our approach preserves the spatial and temporal coherence of spike events, while requiring less computation and memory. The experimental validation shows that our proposed framework outperforms all recent methods on standard datasets. Finally, to address the absence of large realworld NVS datasets for complex recognition tasks, we introduce, evaluate and make available the American Sign Language letters (ASL-DVS), as well as human action dataset (UCF101-DVS, HMDB51-DVS and ASLAN-DVS). Figure 1: Examples of archery action captured by APS and NVS sensors. APS sensors capture images at fixed frame rates, while NVS sensors output a stream of events.

show abstract

Neuromorphic Vision Sensing for CNN-based Action Recognition

Cited by 17 publications

References 17 publications

Graph-Based Spatio-Temporal Feature Learning for Neuromorphic Vision Sensing

Graph-Based Spatio-Temporal Feature Learning for Neuromorphic Vision Sensing

Making Sense of Neuromorphic Event Data for Human Action Recognition

Graph-Based Object Classification for Neuromorphic Vision Sensing

Contact Info

Product

Resources

About