Covariance-based information processing in reservoir computing systems

Abstract

In biological neuronal networks, information representation and processing are achieved through plasticity learning rules that have been empirically characterized as sensitive to second and higher-order statistics in spike trains. However, most models in both computational neuroscience and machine learning aim to convert diverse statistical properties in inputs into first-order statistics in outputs, like in modern deep learning tools. In the context of classification, such schemes have merit for inputs like static images, but they are not well suited to capture the temporal structure in time series. In contrast, the recently developed covariance perceptron uses second-order statistics by mapping input covariances to output covariances in a consistent fashion. Here, we explore the applicability of covariance-based perceptron readouts in reservoir computing networks to classify synthetic multivariate time series structured at different statistical orders (first and second). We show that the second-order framework outperforms or matches the classical mean paradigm in terms of accuracy. We expose nontrivial relationships between input, reservoir and output dynamics, which suggest an important role for recurrent connectivity in transforming information representations in biologically inspired architectures. Finally, we solve a real automatic speech recognition task for the classification of spoken digits to further demonstrate the potential of covariance-based decoding.

Neuronal architecture extracts statistical temporal patterns

“…For neural classifiers that perform at the current state of the art, this may suggest as a metric their flexibility with regard to the adaptive propagation of cumulants. For example, in the spirit of [40] which have showed superior performance of covariance encoding in reservoir computing compared to linear encoding, an order selective perceptron could be combined with a recurrent reservoir to combine the benefits of both. A thorough study of more sophisticated, large-scale architectures that utilize high order cumulants in this regard remains for future work.…”