Echo State Networks-Based Reservoir Computing for MNIST Handwritten Digits Recognition

Schaetti, Nils; Salomon, Michel; Couturier, Raphaël

doi:10.1109/cse-euc-dcabes.2016.229

Cited by 56 publications

(50 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It works particularly well for analyzing time series input data due to its short-term memory [15] and high-dimensional encoding of the input [35,36]. The input images are hence converted into a "time series" by feeding the reservoir a column of the input image at each time point (as in [37]). The method of "temporalization" of the input (row-wise, columnwise, etc.)…”

Section: Methodsmentioning

confidence: 99%

Similarity Learning and Generalization with Limited Data: A Reservoir Computing Approach

2018

View full text Add to dashboard Cite

We investigate the ways in which a machine learning architecture known as Reservoir Computing learns concepts such as “similar” and “different” and other relationships between image pairs and generalizes these concepts to previously unseen classes of data. We present two Reservoir Computing architectures, which loosely resemble neural dynamics, and show that a Reservoir Computer (RC) trained to identify relationships between image pairs drawn from a subset of training classes generalizes the learned relationships to substantially different classes unseen during training. We demonstrate our results on the simple MNIST handwritten digit database as well as a database of depth maps of visual scenes in videos taken from a moving camera. We consider image pair relationships such as images from the same class; images from the same class with one image superposed with noise, rotated 90°, blurred, or scaled; images from different classes. We observe that the reservoir acts as a nonlinear filter projecting the input into a higher dimensional space in which the relationships are separable; i.e., the reservoir system state trajectories display different dynamical patterns that reflect the corresponding input pair relationships. Thus, as opposed to training in the entire high-dimensional reservoir space, the RC only needs to learns characteristic features of these dynamical patterns, allowing it to perform well with very few training examples compared with conventional machine learning feed-forward techniques such as deep learning. In generalization tasks, we observe that RCs perform significantly better than state-of-the-art, feed-forward, pair-based architectures such as convolutional and deep Siamese Neural Networks (SNNs). We also show that RCs can not only generalize relationships, but also generalize combinations of relationships, providing robust and effective image pair classification. Our work helps bridge the gap between explainable machine learning with small datasets and biologically inspired analogy-based learning, pointing to new directions in the investigation of learning processes.

show abstract

Section: Methodsmentioning

confidence: 99%

Similarity Learning and Generalization with Limited Data: A Reservoir Computing Approach

2018

View full text Add to dashboard Cite

show abstract

“…The third approach, inspired by [36], uses an explicit temporal encoding of the spatial visual information from the MNIST images in order to activate the recurrent dynamics of the reservoir computer. The idea here is to split the full image into smaller portions and feed them sequentially into the classifier.…”

Section: Column-wise Recurrent Modementioning

confidence: 99%

“…The separation can be done in various ways: columns or rows (overlapping or adjacent), sliding windows, sub-images, among others. Similarly to [36], we consider nonoverlapping columns, which transforms each image into 28 inputs of 28 dimensions. Therefore, such temporal encoding reduces the input dimensionality, but increases the processing time proportionally.…”

Section: Column-wise Recurrent Modementioning

confidence: 99%

Large-Scale Spatiotemporal Photonic Reservoir Computer for Image Classification

Antonik

Marsal²,

Rontani³

2020

IEEE J. Select. Topics Quantum Electron.

View full text Add to dashboard Cite

We propose a scalable photonic architecture for implementation of feedforward and recurrent neural networks to perform the classification of handwritten digits from the MNIST database. Our experiment exploits off-the-shelf optical and electronic components to currently achieve a network size of 16,384 nodes. Both network types are designed within the the reservoir computing paradigm with randomly weighted input and hidden layers. Using various feature extraction techniques (e.g. histograms of oriented gradients, zoning, Gabor filters) and a simple training procedure consisting of linear regression and winner-takes-all decision strategy, we demonstrate numerically and experimentally that a feedforward network allows for classification error rate of 1%, which is at the state-of-the-art for experimental implementations and remains competitive with more advanced algorithmic approaches. We also investigate recurrent networks in numerical simulations by explicitly activating the temporal dynamics, and predict a performance improvement over the feedforward configuration.

show abstract

“…This function has to be learned by the model in order to solve the task. It would have been possible to use the MNIST database (Schaetti, Salomon, & Couturier, 2016) instead of a regular font but this would have also complexified the task, and make the training period much longer, because a digit is processed only when a trigger is present. If we consider for example a sequence of 25,000 digits and a trigger probability of 0.01, this represents (in average) 250 triggers for the whole sequence and consequently only 25 presentations per digit.…”

Section: The Digit 1-value 1-gate Working Memory Taskmentioning

confidence: 99%

A Robust Model of Gated Working Memory

2020

View full text Add to dashboard Cite

Gated working memory is defined as the capacity of holding arbitrary information at any time in order to be used at a later time. Based on electrophysiological recordings, several computational models have tackled the problem using dedicated and explicit mechanisms. We propose instead to consider an implicit mechanism based on a random recurrent neural network. We introduce a robust yet simple reservoir model of gated working memory with instantaneous updates. The model is able to store an arbitrary real value at random time over an extended period of time. The dynamics of the model is a line attractor that learns to exploit reentry and a nonlinearity during the training phase using only a few representative values. A deeper study of the model shows that there is actually a large range of hyperparameters for which the results hold (e.g., number of neurons, sparsity, global weight scaling) such that any large enough population, mixing excitatory and inhibitory neurons, can quickly learn to realize such gated working memory. In a nutshell, with a minimal set of hypotheses, we show that we can have a robust model of working memory. This suggests this property could be an implicit property of any random population, that can be acquired through learning. Furthermore, considering working memory to be a physically open but functionally closed system, we give account on some counterintuitive electrophysiological recordings.

show abstract

Echo State Networks-Based Reservoir Computing for MNIST Handwritten Digits Recognition

Cited by 56 publications

References 13 publications

Similarity Learning and Generalization with Limited Data: A Reservoir Computing Approach

Similarity Learning and Generalization with Limited Data: A Reservoir Computing Approach

Large-Scale Spatiotemporal Photonic Reservoir Computer for Image Classification

A Robust Model of Gated Working Memory

Contact Info

Product

Resources

About