Adaptive Critic Design with Echo State Network

Koprinkova–Hristova, Petia; Oubbati, Mohamed; Palm, Guenther

doi:10.1109/icsmc.2010.5641744

Cited by 21 publications

(8 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…positively reinforced), the weight adaptation proceeds to finally find a suitable combination between the two learning systems. Previously in [10] [11] an application of the echo-state network (specific RC network) as an adaptive critic for reinforcement learning was presented. Although the authors implemented an online learner, the training and testing data for the RC network were carried out by manually controlling a wheeled robot.…”

Section: Introductionmentioning

confidence: 99%

Neural Combinatorial Learning of Goal-Directed Behavior with Reservoir Critic and Reward Modulated Hebbian Plasticity

Dasgupta

Wörgötter

Morimoto³

et al. 2013

2013 IEEE International Conference on Systems, Man, and Cybernetics

View full text Add to dashboard Cite

Learning of goal-directed behaviors in biological systems is broadly based on associations between conditional and unconditional stimuli. This can be further classified as classical conditioning (correlation-based learning) and operant conditioning (reward-based learning). Although traditionally modeled as separate learning systems in artificial agents, numerous animal experiments point towards their co-operative role in behavioral learning. Based on this concept, the recently introduced framework of neural combinatorial learning combines the two systems where both the systems run in parallel to guide the overall learned behavior. Such a combinatorial learning demonstrates a faster and efficient learner. In this work, we further improve the framework by applying a reservoir computing network (RC) as an adaptive critic unit and reward modulated Hebbian plasticity. Using a mobile robot system for goal-directed behavior learning, we clearly demonstrate that the reservoir critic outperforms traditional radial basis function (RBF) critics in terms of stability of convergence and learning time. Furthermore the temporal memory in RC allows the system to learn partially observable markov decision process scenario, in contrast to a memoryless RBF critic.

show abstract

Section: Introductionmentioning

confidence: 99%

Neural Combinatorial Learning of Goal-Directed Behavior with Reservoir Critic and Reward Modulated Hebbian Plasticity

Dasgupta

Wörgötter

Morimoto³

et al. 2013

2013 IEEE International Conference on Systems, Man, and Cybernetics

View full text Add to dashboard Cite

show abstract

“…Liu et al solved the constrained optimal control problem of unknown discretetime nonlinear systems based on the iterative ADP algorithm via GDHP technique with three neural networks [12]. In fact, different kinds of neural networks (NNs) play the important roles in ADP algorithms, such as radial basis function NNs [3], wavelet basis function NNs [13], and echo state network [14].…”

Section: Introductionmentioning

confidence: 99%

Two-Phase Iteration for Value Function Approximation and Hyperparameter Optimization in Gaussian-Kernel-Based Adaptive Critic Design

Chen

Xie

Xiong

et al. 2015

Mathematical Problems in Engineering

View full text Add to dashboard Cite

Adaptive Dynamic Programming (ADP) with critic-actor architecture is an effective way to perform online learning control. To avoid the subjectivity in the design of a neural network that serves as a critic network, kernel-based adaptive critic design (ACD) was developed recently. There are two essential issues for a static kernel-based model: how to determine proper hyperparameters in advance and how to select right samples to describe the value function. They all rely on the assessment of sample values. Based on the theoretical analysis, this paper presents a two-phase simultaneous learning method for a Gaussian-kernel-based critic network. It is able to estimate the values of samples without infinitively revisiting them. And the hyperparameters of the kernel model are optimized simultaneously. Based on the estimated sample values, the sample set can be refined by adding alternatives or deleting redundances. Combining this critic design with actor network, we present a Gaussian-kernel-based Adaptive Dynamic Programming (GK-ADP) approach. Simulations are used to verify its feasibility, particularly the necessity of two-phase learning, the convergence characteristics, and the improvement of the system performance by using a varying sample set.

show abstract

“…The basic idea of ESN is to use a large number of "reservoirs" as the supplier of some useful dynamics, and the desired output can be combined from the "reservoirs". Recursive least square method is used for the online training [34]. The ESN was proposed by Jaeger [35,36], and developed by many scholars.…”

Section: Introductionmentioning

confidence: 99%

A new self-learning optimal control laws for a class of discrete-time nonlinear systems based on ESN architecture

Song

Xiao

Sun

2014

Sci. China Inf. Sci.

View full text Add to dashboard Cite

A novel self-learning optimal control method for a class of discrete-time nonlinear systems is proposed based on iteration adaptive dynamic programming (ADP) algorithm. It is proven that the iteration costate functions converge to the optimal one, and a detailed convergence analysis of the iteration ADP algorithm is given. Furthermore, echo state network (ESN) architecture is used as the approximator of the costate function for each iteration. To ensure the reliability of the ESN approximator, the ESN mean square training error is constrained in the satisfactory range. Two simulation examples are given to demonstrate that the proposed control method has a fast response speed due to the special structure and the fast training process.

show abstract

Adaptive Critic Design with Echo State Network

Cited by 21 publications

References 11 publications

Neural Combinatorial Learning of Goal-Directed Behavior with Reservoir Critic and Reward Modulated Hebbian Plasticity

Neural Combinatorial Learning of Goal-Directed Behavior with Reservoir Critic and Reward Modulated Hebbian Plasticity

Two-Phase Iteration for Value Function Approximation and Hyperparameter Optimization in Gaussian-Kernel-Based Adaptive Critic Design

A new self-learning optimal control laws for a class of discrete-time nonlinear systems based on ESN architecture

Contact Info

Product

Resources

About