Predicting Performance using Approximate State Space Model for Liquid State Machines

Abstract

Liquid State Machine (LSM) is a brain-inspired architecture used for solving problems like speech recognition and time series prediction. LSM comprises of a randomly connected recurrent network of spiking neurons. This network propagates the non-linear neuronal and synaptic dynamics. Maass et al. have argued that the non-linear dynamics of LSMs is essential for its performance as a universal computer. Lyapunov exponent (µ), used to characterize the non-linearity of the network, correlates well with LSM performance. We propose a complementary approach of approximating the LSM dynamics with a linear state space representation. The spike rates from this model are well correlated to the spike rates from LSM. Such equivalence allows the extraction of a memory metric (τM ) from the state transition matrix. τM displays high correlation with performance. Further, high τM system require lesser epochs to achieve a given accuracy. Being computationally cheap (1800× time efficient compared to LSM), the τM metric enables exploration of the vast parameter design space. We observe that the performance correlation of the τM surpasses the Lyapunov exponent (µ), (2 − 4× improvement) in the high-performance regime over multiple datasets. In fact, while µ increases monotonically with network activity, the performance reaches a maxima at a specific activity described in literature as the edge of chaos. On the other hand, τM remains correlated with LSM performance even as µ increases monotonically. Hence, τM captures the useful memory of network activity that enables LSM performance. It also enables rapid design space exploration and fine-tuning of LSM parameters for high performance.

Hardware-Friendly Synaptic Orders and Timescales in Liquid State Machines for Speech Classification

“…Recently, a digital implementation of an LSM using these models was proposed for the TI-46 spoken digits recognition task with a spike based local learning rule for the linear classifier [17]. Further, the LSM network was represented using a statespace model and a performance predicting memory metric was extracted [11]. It has been argued that the post synaptic current waveform has a crucial role to play in the classification accuracy [18]- [20] specifically for the speech digit recognition task [17] (Table I).…”

“…channels for each input digit sample. The details of the preprocessing stage have been discussed in detail elsewhere [11], [24]. These 500 samples are used for training and testing the network in a 5-fold manner.…”

“…The reservoir consists of 125 neurons. The reservoir connectivity (𝑮 𝑟𝑒𝑠 ) is established using the local probabilistic random connections model with 80% excitatory neurons and 20% inhibitory neurons [11]. This model requires the neurons to be arranged on a grid (5 X 5 X 5 chosen here) to calculate the connection probability as a function of the distance between the neurons in the grid structure.…”

P-CRITICAL: a reservoir autoregulation plasticity rule for neuromorphic hardware

“…The first and most straightforward category is simply post training accuracy. The second type of heuristic consists of a posteriori methods [4][5][6][7] that require some simulation time without the need for training the network on a dataset, e.g., computing the separation distance of closely inputs using the Lyapunov exponent [4]. Lastly, the a priori category encompasses methods that can create reservoirs without the need of simulation [8,9] by algorithmically creating the reservoir from a mathematical definition.…”

“…Other metrics have been explored to quantify performance based on a posteriori dynamic analysis such as the Lyapunov exponent [4], the average state entropy [5], the dynamic profile of the Jacobian of W R [6] or the approximate state space model [7]. These methods allow for a more guided search on a reservoir's parameters.…”