Learning automata based multi‐agent system algorithms for finding optimal policies in Markov games

Abstract

Markov games, as the generalization of Markov decision processes to the multi agent case, have long been used for modeling multi-agent systems. The Markov game view of MAS is considered as a sequence of games having to be played by multiple players while each game belongs to a different state of the environment. In this paper, several learning automata based multi-agent system algorithms for finding optimal policies in Markov games are proposed. In all of the proposed algorithms, each agent residing in every state of the environment is equipped with a learning automaton. Every joint-action of the set of learning automata in each state corresponds to moving to one of the adjacent states. Each agent moves from one state to another and tries to reach the goal state. The actions taken by learning automata along the path traversed by the agent are then rewarded or penalized based on the comparison of the average reward received by agent per move along the path with a dynamic threshold. In the second group of the proposed algorithms, the concept of entropy has been imported into learning automata based multi-agent systems to improve the performance of the algorithms. To evaluate the performance of the proposed algorithms, computer experiments have been conducted. The results of experiments have shown that the proposed algorithms perform better than the existing algorithms in terms of speed and accuracy of reaching the optimal policy. Keywords: Markov games, Multi-Agent Systems, Learning Automata, Optimal Policy 1-INTRODUCTIONThere are several models proposed in the literatures for multi-agent systems (MAS) based on Markov models. One of these models is Markov Game (MG) which is the generalization of the Markov decision process (MDP)to the Multiple agent [1]. The Markov game view of MAS is considered as a sequence of games having to be played by multiple players while each game belongs to a different state of the environment [2]. In a MG, actions are the result of joint action selection of all agents, while rewards and the state transitions depend on these joint actions [3]. As a special case, when only is one state assumed, the Markov game is known as a repeated normal form game in game theory [1]. In addition, when only is one agent assumed, the Markov game is known as MDP. In multi-agent system research, two main perspectives are found in the literature; the cooperative and non-cooperative perspective. In cooperative MASs, the agents pursue a common goal and the agents can be built expect benevolent intentions from other agents [4]. In contrast, a non-cooperative MAS setting has non-aligned goals, and individual agents try to obtain only to maximize their own profits. In multi-agent systems, the need for learning and adaption is essentially caused by the fact that the environment of agent is dynamic and just empirically observable while the environment (the reward functions and the transition states) is unknown. Hence, the reinforcement learning methods may be applied in MAS to find an optimal policy in MG...

Using Learning Automata to Solve Dynamic Distributed Constraint Optimization Problems

“…The use of LA and its extension in the context of Markov games to solve distributed Markov decision-making problems has been investigated in studies [40,52]. In these methods, each agent in each state of the environment is equipped with a learning automaton.…”

Continuous Action Generation of Q‐Learning in Multi‐Agent Cooperation

“…Although (4) can be extended to general-sum SGs, minimax-Q is no longer well-motivated in those settings. In our problem, the two agents have to cooperate rather than compete with each other.…”

“…In recent years, reinforcement learning (RL) has become an attractive learning method for training robots . More and more research projects have extended the reinforcement learning methods to multi‐agent systems , where agents know little about other agents and the environment changes during learning. Applications of reinforcement learning in multi‐agent systems include soccer , pursuit games , and coordination games .…”

Heuristic Dynamic Programming Using Echo State Network For Multivariable Tracking Control Of Wastewater Treatment Process

“…For a class of unknown general nonlinear systems such as WWTP, adaptive dynamic programming (ADP)-also known as reinforcement learning, has attracted more and more attention of experts recently [8][9][10][11][12][13][14][15][16][17][18][19][20][21][22][23][24]. The core of ADP is the approximation of Bellman's equation or the Hamilton-Jacobi-Bellman (HJB) equation by learning [10,12,20].…”

“…Most ADP schemes use artificial neural networks (ANNs) [13][14][15][16][17][18][19][20]. A recurrent neural network (RNN) appears as a highly promising and fascinating tool for nonlinear dynamics control application because of its universal approximating ability to dynamical systems [25].…”