Decentralized Multi-Agent Reinforcement Learning in Average-Reward Dynamic DCOPs

Abstract

Researchers have introduced the Dynamic Distributed Constraint Optimization Problem (Dynamic DCOP) formulation to model dynamically changing multi-agent coordination problems, where a dynamic DCOP is a sequence of (static canonical) DCOPs, each partially different from the DCOP preceding it. Existing work typically assumes that the problem in each time step is decoupled from the problems in other time steps, which might not hold in some applications. Therefore, in this paper, we make the following contributions: (i) We introduce a new model, called Markovian Dynamic DCOPs (MD-DCOPs), where the DCOP in the next time step is a function of the value assignments in the current time step; (ii) We introduce two distributed reinforcement learning algorithms, the Distributed RVI Q-learning algorithm and the Distributed R-learning algorithm, that balance exploration and exploitation to solve MD-DCOPs in an online manner; and (iii) We empirically evaluate them against an existing multi-arm bandit DCOP algorithm on dynamic DCOPs.

Using Learning Automata to Solve Dynamic Distributed Constraint Optimization Problems

“…The Markovian DDCOPs algorithm [8] is a reactive online learning based on the joint action of agents. Unlike other methods, the proactive algorithms [9,27] consider the dependence between two consecutive time steps.…”

“…𝜎: 𝑋 → 𝐴 is an onto total function, from variables to agents, which assigns the control of each variable 𝑥 ∈ 𝑋 to an agent 𝜎(𝑥). A complete solution is a value assignment for all variables and the objective is to find a reward-maximal complete solution [8].…”

“…For coordinate situations such as agents in complicated, dynamic and unspecified environments, agents coordinate together. In dynamic problems, agents may face with different problems, new agents may be added to the system or new information become available and flow of information can change the environment, so researchers introduced dynamic DCOPs [8].…”

“…In these researches, agents have holistic information about current environment (current DCOP) but are unaware of future changes of information [9]. So, ignores the dependence of each problem among each time steps and can't make use of information of previous DCOPs [8]. They solve current problem using conventional DCOP method and if there be a need for change in the problem, they search for a new cue, this will need more time and memory [10].…”

Efficient Average Reward Reinforcement Learning Using Constant Shifting Values

“…Average reward MDPs are natural models of many non-terminating tasks, such as the call admission control and routing problem (Marbach, Mihatsch, and Tsitsiklis 2000) and the automatic guided vehicle routing problem (Ghavamzadeh and Mahadevan 2007). Average reward RL has received much attention in the recent years (Ortner 2013;Mahadevan 2014;Nguyen et al 2014).…”