Effects of singular value spectrum on the performance of echo state network

Li, Fanjun; Wang, Xiaohong; Liu, Ying

doi:10.1016/j.neucom.2019.05.068

Cited by 10 publications

(4 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Reservoir computing is grounded in the idea of using a large randomly and sparsely connected recurrent layer called a reservoir. This method presents an efficient alternative to gradient-based learning algorithms for designing and training RNNs in most cases [ 49 ]. The echo state network (ESN) and its variants have stimulated researchers’ interest in recent decades owing to simple model training.…”

Section: Introductionmentioning

confidence: 99%

Ensemble and Pre-Training Approach for Echo State Network and Extreme Learning Machine Models

Tang,

Wang,

Wang

et al. 2024

Entropy

View full text Add to dashboard Cite

The echo state network (ESN) is a recurrent neural network that has yielded state-of-the-art results in many areas owing to its rapid learning ability and the fact that the weights of input neurons and hidden neurons are fixed throughout the learning process. However, the setting procedure for initializing the ESN’s recurrent structure may lead to difficulties in designing a sound reservoir that matches a specific task. This paper proposes an improved pre-training method to adjust the model’s parameters and topology to obtain an adaptive reservoir for a given application. Two strategies, namely global random selection and ensemble training, are introduced to pre-train the randomly initialized ESN model. Specifically, particle swarm optimization is applied to optimize chosen fixed and global weight values within the network, and the reliability and stability of the pre-trained model are enhanced by employing the ensemble training strategy. In addition, we test the feasibility of the model for time series prediction on six benchmarks and two real-life datasets. The experimental results show a clear enhancement in the ESN learning results. Furthermore, the proposed global random selection and ensemble training strategies are also applied to pre-train the extreme learning machine (ELM), which has a similar training process to the ESN model. Numerical experiments are subsequently carried out on the above-mentioned eight datasets. The experimental findings consistently show that the performance of the proposed pre-trained ELM model is also improved significantly. The suggested two strategies can thus enhance the ESN and ELM models’ prediction accuracy and adaptability.

show abstract

Section: Introductionmentioning

confidence: 99%

Ensemble and Pre-Training Approach for Echo State Network and Extreme Learning Machine Models

Tang,

Wang,

Wang

et al. 2024

Entropy

View full text Add to dashboard Cite

show abstract

“…It has conventionally been chosen to be slightly less than 1. Moreover, the weight matrix in the reservoir layer W res is scaled by the spectral radius in order to balance the validity of the ESP and the performance according to [26,27]. Lastly, the resulting output response at time t can be described by…”

Section: Introductionmentioning

confidence: 99%

Novel efficient reservoir computing methodologies for regular and irregular time series classification

Li,

Andreev,

Hramov

et al. 2024

Preprint

View full text Add to dashboard Cite

Time series is a data structure prevalent in a wide range of fields such as healthcare, finance and meteorology. Analyzing time series data holds the key to gaining insight into our day-to-day observations and among them, time series classification offers the unique opportunity to classify the sequences into their respective categories for the sake of automated detection. To this end, two types of mainstream approaches, recurrent neural networks and distance-based methods, have been commonly employed to address this specific problem. However, the most successful ones such as Long Short-Term Memory networks generally suffer substantially high computational demand, prompting the search for more efficient alternatives to reduce energy costs. Reservoir computing is an instance of recurrent neural networks that is known for its efficiency in processing time series sequences. Therefore, in this article, we will develop two reservoir computing based methods that can effectively deal with time series of different types with minimal computational cost, all while achieving a desirable level of classification accuracy. Mathematics Subject Classification (2020) 34A34 · 62P10 · 68T07

show abstract

“…Steiner et al [11] used the Kmeans algorithm for unsupervised initialization of the input weight matrix, which made the network performance comparable or better than that of a randomly initialized ESN, while requiring significantly fewer reservoir neurons. Li et al [12] designed reservoirs with predefined sparsity and singular values. Chen et al [13] constructed an incremental inverse-free ESN to obtain weights according to the information of the previous optimal reservoir state and the freshly added neurons in the reservoir.…”

Section: Introductionmentioning

confidence: 99%

Semi-supervised Echo State Network with Partial Correlation Pruning for Time-Series Variables Prediction in Industrial Processes

Huang¹,

Wang²,

Yang³

et al. 2023

Meas. Sci. Technol.

View full text Add to dashboard Cite

For an ordinary echo state network (ESN), redundant information in the huge reservoir will lead to degradation of the prediction performance of the network, especially when the labels of the samples are limited. To solve this problem, a semi-supervised ESN with partial correlation pruning (PCP-S2ESN) is proposed in this paper to scientifically capture the essential association between two reservoir variables while controlling for the influence of other factors. In this way, redundant neurons and their connection weights in the reservoir are eliminated, so that the prediction accuracy is significantly enhanced by optimizing the network structure. Moreover, anunsupervised pre-training procedure is introduced to modify the input weight matrix and reservoir connection weight matrix of the ESN, which successfully achieves precise prediction of time-series variables with limited labels. The superiority of the PCP-S2ESN model is demonstrated through two benchmark prediction tasks and the fed-batch penicillin cultivation (FBPC) process.

show abstract

Effects of singular value spectrum on the performance of echo state network

Cited by 10 publications

References 39 publications

Ensemble and Pre-Training Approach for Echo State Network and Extreme Learning Machine Models

Ensemble and Pre-Training Approach for Echo State Network and Extreme Learning Machine Models

Novel efficient reservoir computing methodologies for regular and irregular time series classification

Semi-supervised Echo State Network with Partial Correlation Pruning for Time-Series Variables Prediction in Industrial Processes

Contact Info

Product

Resources

About