A Markov Game-Adaptive Fuzzy Controller for Robot Manipulators

Abstract

This paper develops an adaptive fuzzy controller for robot manipulators using a Markov game formulation. The Markov game framework offers a promising platform for robust control of robot manipulators in the presence of bounded external disturbances and unknown parameter variations. We propose fuzzy Markov games as an adaptation of fuzzy Q-learning (FQL) to a continuous-action variation of Markov games, wherein the reinforcement signal is used to tune online the conclusion part of a fuzzy Markov game controller. The proposed Markov game-adaptive fuzzy controller uses a simple fuzzy inference system (FIS), is computationally efficient, generates a swift control, and requires no exact dynamics of the robot system. To illustrate the superiority of Markov game-adaptive fuzzy control, we compare the perfor- mance of the controller against a) the Markov game-based robust neural controller, b) the reinforcement learning (RL)-adaptive fuzzy controller, c) the FQL controller, d) the theory-based robust neural game controller, and e) a standard RL-based robust neural controller, on two highly nonlinear robot arm control problems of i) a standard two-link rigid robot arm and ii) a 2-DOF SCARA robot manipulator. The proposed Markov game-adaptive fuzzy controller outperformed other controllers in terms of tracking errors and control torque requirements, over different desired trajectories. The results also demonstrate the viability of FISs for accelerating learning in Markov games and extendingMarkov game-based control to continuous state-action space problems.Index Terms-Fuzzy Q-learning (FQL), Markov game-adaptive fuzzy controller, Markov games, SCARA robot, two-link robot arm.

Synergizing reinforcement learning and game theory—A new direction for control

“…In spite of some benchmark implementations of Markov games in conjunction with function approximation, showing convergence or establishing error bounds, as also in MDP based RL, remains an open problem. We emphasize that function approximation represents a principled means for scaling to large or continuous state/state-action space problems [56].…”

“…In [56], we proposed fuzzy Markov games as an adaptation of fuzzy Q-learning to a continuous-action variation of Markov games, wherein the reinforcement signal is used to tune online conclusion part of a fuzzy Markov game controller. Markov game adaptive fuzzy controller (MGAFC) architecture is similar to standard RLC [62] and RLAFC [53], wherein controller is a combination of an action-generating NN/FIS that uses a RL signal as an adaptivelearning signal, and a fixed gain controller in the PD loop.…”

“…Markov game adaptive fuzzy controller [56] tunes only the consequent part of FIS, i.e., FIS (B). FIS (B) (rule-base consequent part), generates a q-value corresponding to each rule R i , controller action and disturber action, i.e., it generates q(i,u i ,d i )∀u i ∈ U, d i ∈ D. These values are used to form the game matrix at current state x k , for each rule R i .…”

Multi-layer Neural Network for Servo Motor Control

“…A survey conducted has shown researchers adopt fuzzy logic [16] for the control but the results discussed in the paper lack accuracy of control. The use of neuro fuzzy logic proposed by researchers in [17] [18] demonstrated good learning capabilities, enabled to address the issue of a large torque ripple at low operating speeds but could not reduce the computational complexity of the overall system. The open research issues that exist are clearly discussed in [17] [18] [19].A comprehensive literature review of the techniques proposed by researcher's in the parameter control of motor drives is presented in [20].…”

SCARA Robot: Modeled, Simulated, and Virtual-Reality Verified

“…A simulated robotic setup costs less than do real robots and real-world setups, and designs can be better explored. Simulation often is faster than the real movement, and parameters are visible on screen [1][2][3][4][5].…”