Fuzzy Sarsa Learning and the proof of existence of its stationary points

Abstract

This paper provides a new Fuzzy Reinforcement Learning (FRL) algorithm based on critic‐only architecture. The proposed algorithm, called Fuzzy Sarsa Learning (FSL), tunes the parameters of conclusion parts of the Fuzzy Inference System (FIS) online. Our FSL is based on Sarsa, which approximates the Action Value Function (AVF) and is an on‐policy method. In each rule, actions are selected according to the proposed modified Softmax action selection so that the final inferred action selection probability in FSL is equivalent to the standard Softmax formula. We prove the existence of fixed points for the proposed Approximate Action Value Iteration (AAVI). Then, we show that FSL satisfies the necessary conditions that guarantee the existence of stationary points for it, which coincide with the fixed points of the AAVI. We prove that the weight vector of FSL with stationary action selection policy converges to a unique value. We also compare by simulation the performance of FSL and Fuzzy Q‐Learning (FQL) in terms of learning speed, and action quality. Moreover, we show by another example the convergence of FSL and the divergence of FQL when both algorithms use a stationary policy. Copyright © 2008 John Wiley and Sons Asia Pte Ltd and Chinese Automatic Control Society

Dynamic fuzzy logic and reinforcement learning for adaptive energy efficient routing in mobile ad-hoc networks

“…In this latter, the best action is given the highest selection probability; whereas, all remaining actions are ordered in function of their estimated Q values. In particular, a modified Boltzmann softmax action selection rule proposed in [36] is applied. In rule i, action a ij is chosen with probability: Table 2 Fuzzy If-Then mapping rules for choosing RREQs forwarding probability in FSARSA-AODV.…”

Demonstration of Learned Helplessness with Fuzzy Reinforcement Learning

“…In this section, we describe FSL briefly; readers can find the comprehensive information about FSL in [4].…”

“…Notice that to calculate the overall action, first an action is selected for each rule from among the candidate actions of that rule. Denoting the selected action in i-th rule and its corresponding value by  ii a and  ii w , respectively, the system output (i.e., the overall continuous action) and its corresponding approximate Action Value Function (AVF) are computed as follows [4,7]: w . Then, the weight parameters of the i-th rule are updated by [4]:…”

“…The action selection probability of the i-th candidate action in the i-th rule in state t s is computed based on the following modified Softmax policy [4]:…”

“…Here, we focus on Fuzzy Sarsa learning (FSL) that is the first critic-only FRL with mathematical analysis [4]. This algorithm tunes the parameters of conclusion parts of the fuzzy system rules online.…”