HyNNA: Improved Performance for Neuromorphic Vision Sensor Based Surveillance using Hybrid Neural Network Architecture

Abstract

Applications in the Internet of Video Things (IoVT) domain have very tight constraints with respect to power and area. While neuromorphic vision sensors (NVS) may offer advantages over traditional imagers in this domain, the existing NVS systems either do not meet the power constraints or have not demonstrated end-to-end system performance. To address this, we improve on a recently proposed hybrid event-frame approach by using morphological image processing algorithms for region proposal and address the low-power requirement for object detection and classification by exploring various convolutional neural network (CNN) architectures. Specifically, we compare the results obtained from our object detection framework against the state-of-the-art low-power NVS surveillance system and show an improved accuracy of 82.16% from 63.1%. Moreover, we show that using multiple bits does not improve accuracy, and thus, system designers can save power and area by using only single bit event polarity information. In addition, we explore the CNN architecture space for object classification and show useful insights to trade-off accuracy for lower power using lesser memory and arithmetic operations.

e-TLD: Event-Based Framework for Dynamic Object Tracking

“…Using the DAVIS, [27]- [29] use a convolutional neural network (CNN) to detect likely target locations for tracking from a moving platform. However, a hybrid approach with frames and events naturally loses the advantages of a low-latency, purely event-driven approach, although can provide energy savings in hardware implementations [30]. On the other hand, [31] uses a parametric model to motion-compensate for the camera, without explicit feature tracking or optical flow computation, and subsequently moving objects that do no confirm to the model are detected in an iterative fashion.…”

A 51.3 TOPS/W, 134.4 GOPS In-memory Binary Image Filtering in 65nm CMOS

“…2) Object Classification: To assess the effect of approximation of NOMF on object classification, we have also classified vehicles into 4 categories (Car, Bus, Track/Van and Bike) from the above traffic dataset following [34]. Objects in each frame are manually labelled and also the same objects across frames are assigned the same track identifier-this comprises the ground truth.…”

“…The 42 × 42 patches are filtered using both conventional median filter (OMF) and NOMF to produce two different datasets for the same LeNet5 model. For a fair comparison, the same LeNet5 model is trained and tested separately using 91885 and 9063 filtered images from the conventional median filter and NOMF, respectively, following the methodology used in [34].…”

“…Table III shows the per sample and per track accuracy for all test samples across different categories. To estimate per track accuracy, the final category is obtained by voting across the classification scores for all the images in a track across multiple frames [34]. We use balanced accuracy as a metric to overcome the imbalance in number of test samples across classes.…”