Deep Liquid State Machines With Neural Plasticity for Video Activity Recognition

Abstract

Real-world applications such as first-person video activity recognition require intelligent edge devices. However, size, weight, and power constraints of the embedded platforms cannot support resource intensive state-of-the-art algorithms. Machine learning lite algorithms, such as reservoir computing, with shallow 3-layer networks are computationally frugal as only the output layer is trained. By reducing network depth and plasticity, reservoir computing minimizes computational power and complexity, making the algorithms optimal for edge devices. However, as a trade-off for their frugal nature, reservoir computing sacrifices computational power compared to state-of-the-art methods. A good compromise between reservoir computing and fully supervised networks are the proposed deep-LSM networks. The deep-LSM is a deep spiking neural network which captures dynamic information over multiple time-scales with a combination of randomly connected layers and unsupervised layers. The deep-LSM processes the captured dynamic information through an attention modulated readout layer to perform classification. We demonstrate that the deep-LSM achieves an average of 84.78% accuracy on the DogCentric video activity recognition task, beating state-of-the-art. The deep-LSM also shows up to 91.13% memory savings and up to 91.55% reduction in synaptic operations when compared to similar recurrent neural network models. Based on these results we claim that the deep-LSM is capable of overcoming limitations of traditional reservoir computing, while maintaining the low computational cost associated with reservoir computing.

Opportunities for neuromorphic computing algorithms and applications

“…Several demonstrations of spike-based reservoir computing have shown their effectiveness at processing temporally varying signals [52][53][54] . Variants of this computing framework have ranged from simple reservoir networks for bio-signal processing and prosthetic control applications 52 to using hierarchical layers of liquid state machines-a type of reservoir network-interconnected with layers trained in supervised mode for video 55 and audio signal processing applications 54 .…”

Universal Self-Correcting Computing with Disordered Exciton-Polariton Neural Networks

“…The need for control of the internode connections is completely lifted and the network function is rather determined by varying a set of weight connections in a single output layer, which connects the neurons in the network to a final result. Hardware implementations of reservoir computers [31] have been based on memristor arrays [32,33], photonic arrays on silicon chips [34], microwaves [35], electronic circuits [36], and nonlinear optical elements coupled to delay lines [37][38][39][40]. Reservoir computing was recently considered theoretically for polariton systems, where relatively high success rates for standard benchmark tasks such as character recognition were predicted (95% success rate for recognition of the MNIST set of hand-written digits) [19].…”

A human vision based system for biometric images recognition

“…Furthermore, the LSM bears many similarities to biological neural networks and can therefore profit from new findings in neurobiology. It has been used for various real world applications such as movement prediction [5], [6], speech recognition [7]- [9], video activity and image recognition [10]- [12] and was particularly successful used in spatio-temporal pattern recognition [13]- [16].…”