Approximate dynamic programming with a fuzzy parameterization

Abstract

Dynamic programming (DP) is a powerful paradigm for general, nonlinear optimal control. Computing exact DP solutions is in general only possible when the process states and the control actions take values in a small discrete set. In practice, it is necessary to approximate the solutions. Therefore, we propose an algorithm for approximate DP that relies on a fuzzy partition of the state space, and on a discretization of the action space. This fuzzy Q-iteration algorithm works for deterministic processes, under the discounted return criterion. We prove that fuzzy Q-iteration asymptotically converges to a solution that lies within a bound of the optimal solution. A bound on the suboptimality of the solution obtained in a finite number of iterations is also derived. Under continuity assumptions on the dynamics and on the reward function, we show that fuzzy Q-iteration is consistent, i.e., that it asymptotically obtains the optimal solution as the approximation accuracy increases. These properties hold both when the parameters of the approximator are updated in a synchronous fashion, and when they are updated asynchronously. The asynchronous algorithm is proven to converge at least as fast as the synchronous one. The performance of fuzzy Q-iteration is illustrated in a two-link manipulator control problem.

Optimistic planning for sparsely stochastic systems

“…, 15π} Since an exact optimal solution for the inverted pendulum problem is not known, in order to approximate the regret (3), a near-optimal solution is computed instead. To this end, the fuzzy Q-iteration algorithm [7] is modified to work for the sparsely stochastic systems considered in this paper, and applied to the inverted pendulum using a very accurate approximator over the state space. Figure 3, top reports the (approximate) regret of the three algorithms, averaged over the set X 0 .…”

Optimal Control via Reinforcement Learning with Symbolic Policy Approximation

“…. , N. This value iteration is guaranteed to converge (Buşoniu et al, 2010) and terminates when the following condition is satisfied:…”

An Intelligent Strategy Based on Fuzzy Reinforcement Learning for Backing Truck up

“…It can also be simply combined with fuzzy logic and provide the relationship between the states and the accessible action, which is the same as creating the fuzzy logic "if. ..then" engine [23][24][25].…”