Online Tuning of Fuzzy Inference Systems Using Dynamic Fuzzy Q-Learning

Abstract

This paper presents a dynamic fuzzy Q-learning (DFQL) method that is capable of tuning fuzzy inference systems (FIS) online. A novel online self-organizing learning algorithm is developed so that structure and parameters identification are accomplished automatically and simultaneously based only on Q-learning. Self-organizing fuzzy inference is introduced to calculate actions and Q-functions so as to enable us to deal with continuous-valued states and actions. Fuzzy rules provide a natural mean of incorporating the bias components for rapid reinforcement learning. Experimental results and comparative studies with the fuzzy Q-learning (FQL) and continuous-action Q-learning in the wall-following task of mobile robots demonstrate that the proposed DFQL method is superior.

Quantum Reinforcement Learning

“…For example, Smith [16] presented a new model for representation and generalization in model-less RL based on the self-organizing map (SOM) and standard Q-learning. The adaptation of Watkins' Q-learning with fuzzy inference systems for problems with large state-action spaces or with continuous state spaces is also proposed [6], [17], [18], [19]. Many specific improvements are also implemented to modify related RL methods in practice [7], [9], [10], [20], [21], [22].…”

An Approach to Tune PID Fuzzy Logic Controllers Based on Reinforcement Learning

“…If the TD error is positive, it suggests that the quality of this action should be strengthened for future use, whereas if the TD error is negative, it suggests that the quality should weakened [4]. The learning rule by taking the eligibity traces is given by …”

“…An important number of choices is given a priori, these choices are carried with empirical methods, and then the design of the FLC can prove to be long and delicate towards the important number of parameters to determine, and can lead then to a solution with poor performance [12]. With this subjective approach, it is difficult for a designer to examine complex systems to find the necessary number of rules, and to determine appropriate parameters of the rules for implementing the fuzzy controller [13]. Also, it isn't easy to design an optimized fuzzy controller.…”

“…Structure identification, such as partitioning the input and output space and determination of the number of fuzzy rules are still carried out offline and it is time consuming. In [4] a novel online self-organizing learning algorithm is developed so that structure and parameters identification are accomplished automatically and simultaneously based only on Q-learning. In [45], [48], a dynamic fuzzy Q-learning is proposed for fuzzy inference system design.…”

“…In this method, the consequent parts of fuzzy rules are randomly generated and the best rule set is selected based on its corresponding Q-value based genetic reinforcement learning. The problem in these approachs [4], [45], [50] is that if the optimal solution is not present in the randomly generated set, then the performance may be poor. In order to solve these problems, this paper provides a systematic procedure for designing Fuzzy PID (FPID) controllers based on a reinforcement learning method.…”