A decentralized training algorithm for Echo State Networks in distributed big data applications

Scardapane, Simone; Wang, Dianhui; Panella, Massimo

doi:10.1016/j.neunet.2015.07.006

Cited by 109 publications

(56 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We propose, instead, to use information derived from RR (8), DET (9), LAM (12), ENTR (13), and SWRP (14) to determine the edge of stability. In fact, we observe that, when such indices start to fluctuate, the network achieves high prediction accuracy.…”

Section: B Design Of a Stable And Effective Networkmentioning

confidence: 99%

“…It is possible to rely on specific RQA measures, such as DET (9), ENTR (13), and SWRP (14), to numerically quantify the amount of time dependence. All three indices would yield very low values (close to zero) when there is no time-dependence in the signal.…”

Section: Visualization and Classification Of Reservoir Dynamicsmentioning

confidence: 99%

“…Nonetheless, this choice allows the network designer to immediately recognize areas of interest and transitions in the related plots. We always use 50 bins for computing the SWRP measure (14). For each configuration of ρ and ω i , we perform 15 different simulations with independent random initialization of W r i and W r r .…”

Section: B Characterization Of Reservoirs With Rqamentioning

confidence: 99%

“…ESN reservoir characterization and design attracted significant research efforts in the last decade [11]- [13]. This is mostly due to the puzzling behavior of the reservoir, which, although randomly initialized, has shown to be effective in modeling nonlinear dynamical systems of various nature [14]- [18]. This also motivated researchers to better understand, and hence control, the dynamics of reservoirs when they are driven by input signals.…”

mentioning

confidence: 99%

See 3 more Smart Citations

Investigating Echo-State Networks Dynamics by Means of Recurrence Analysis

Bianchi

Livi

Alippi

2018

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

Abstract-In this paper, we elaborate over the well-known interpretability issue in echo-state networks (ESNs). The idea is to investigate the dynamics of reservoir neurons with timeseries analysis techniques developed in complex systems research. Notably, we analyze time series of neuron activations with recurrence plots (RPs) and recurrence quantification analysis (RQA), which permit to visualize and characterize highdimensional dynamical systems. We show that this approach is useful in a number of ways. First, the 2-D representation offered by RPs provides a visualization of the high-dimensional reservoir dynamics. Our results suggest that, if the network is stable, reservoir and input generate similar line patterns in the respective RPs. Conversely, as the ESN becomes unstable, the patterns in the RP of the reservoir change. As a second result, we show that an RQA measure, called L max , is highly correlated with the well-established maximal local Lyapunov exponent. This suggests that complexity measures based on RP diagonal lines distribution can quantify network stability. Finally, our analysis shows that all RQA measures fluctuate on the proximity of the so-called edge of stability, where an ESN typically achieves maximum computational capability. We leverage on this property to determine the edge of stability and show that our criterion is more accurate than two well-known counterparts, both based on the Jacobian matrix of the reservoir. Therefore, we claim that RPs and RQA-based analyses are valuable tools to design an ESN, given a specific problem.Index Terms-Dynamics, echo-state network (ESN), recurrence plot (RP), recurrence quantification analysis (RQA), nonlinear time series analysis.

show abstract

Section: B Design Of a Stable And Effective Networkmentioning

confidence: 99%

Section: Visualization and Classification Of Reservoir Dynamicsmentioning

confidence: 99%

Section: B Characterization Of Reservoirs With Rqamentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

Investigating Echo-State Networks Dynamics by Means of Recurrence Analysis

Bianchi

Livi

Alippi

2018

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

show abstract

“…Among them, we can cite distributed protocols for support vector machines [7], functional-link networks [8], linear neurons [9], adaptive resonance theory (ART) networks [10], and many others. However, as we argued in [11], what is needed in many contexts is a distributed training algorithm for recurrent neural networks (RNNs). Thanks to the presence of recurrent connections, RNNs are able to efficiently capture the dynamics in the underlying process to be learned.…”

Section: Introductionmentioning

confidence: 99%

Distributed Reservoir Computing with Sparse Readouts [Research Frontier]

Scardapane

Panella

Comminiello

et al. 2016

IEEE Comput. Intell. Mag.

Self Cite

View full text Add to dashboard Cite

In a network of agents, a widespread problem is the need to estimate a common underlying function starting from locally distributed measurements. Real-world scenarios may not allow the presence of centralized fusion centers, requiring the development of distributed, message-passing implementations of the standard machine learning training algorithms. In this paper, we are concerned with the distributed training of a particular class of recurrent neural networks, namely echo state networks (ESNs). In the centralized case, ESNs have received considerable attention, due to the fact that they can be trained with standard linear regression routines. Based on this observation, in our previous work we have introduced a decentralized algorithm, framed in the distributed optimization field, in order to train an ESN. In this paper, we focus on an additional sparsity property of the output layer of ESNs, allowing for very efficient implementations of the resulting networks. In order to evaluate the proposed algorithm, we test it on two well-known prediction benchmarks, namely the Mackey-Glass chaotic time series and the 10th order nonlinear auto regressive moving average (NARMA) system.

show abstract

Randomness in neural networks: an overview

Scardapane

Wang

2017

WIREs Data Min & Knowl

Self Cite

231

144

View full text Add to dashboard Cite

Neural networks, as powerful tools for data mining and knowledge engineering, can learn from data to build feature-based classifiers and nonlinear predictive models. Training neural networks involves the optimization of nonconvex objective functions, and usually, the learning process is costly and infeasible for applications associated with data streams. A possible, albeit counterintuitive, alternative is to randomly assign a subset of the networks' weights so that the resulting optimization task can be formulated as a linear least-squares problem. This methodology can be applied to both feedforward and recurrent networks, and similar techniques can be used to approximate kernel functions. Many experimental results indicate that such randomized models can reach sound performance compared to fully adaptable ones, with a number of favorable benefits, including (1) simplicity of implementation, (2) faster learning with less intervention from human beings, and (3) possibility of leveraging overall linear regression and classification algorithms (e.g., ℓ 1 norm minimization for obtaining sparse formulations). This class of neural networks attractive and valuable to the data mining community, particularly for handling large scale data mining in real-time. However, the literature in the field is extremely vast and fragmented, with many results being reintroduced multiple times under different names. This overview aims to provide a self-contained, uniform introduction to the different ways in which randomization can be applied to the design of neural networks and kernel functions. A clear exposition of the basic framework underlying all these approaches helps to clarify innovative lines of research, open problems, and most importantly, foster the exchanges of well-known results throughout different communities.

show abstract

A decentralized training algorithm for Echo State Networks in distributed big data applications

Cited by 109 publications

References 46 publications

Investigating Echo-State Networks Dynamics by Means of Recurrence Analysis

Investigating Echo-State Networks Dynamics by Means of Recurrence Analysis

Distributed Reservoir Computing with Sparse Readouts [Research Frontier]

Randomness in neural networks: an overview

Contact Info

Product

Resources

About