Distributed on-Line Multi-Agent Optimization Under Uncertainty: Balancing Exploration and Exploitation

Abstract

to our expectations, we found that increasing teamwork in DCEE algorithms may lower team performance. In contrast, agents running DCOP algorithms improve their reward as teamwork increases. We term this previously unknown phenomenon the team uncertainty penalty, analyze it in both simulation and on robots, and present techniques to ameliorate the penalty.

Distributed learning and multi-objectivity in traffic light control

“…Furthermore, it is clear that incorporating the current state of the traffic in the decision making process allows for much finer control and response to fluctuations in the general traffic pattern. Also, applying algorithms from the DCEE framework, our previous results concerning the team uncertainty penalty are confirmed in this setting (Taylor et al, 2011), showing again that more coordination among agents is not necessarily beneficial.…”

“…Larger k values allows for more joint moves. This often, but not always, increases the total team performance (Taylor et al, 2011). This paper focuses on the class of static estimation (SE) DCEE algorithms.…”

“…K2 performs worse than K1, even though it involves more coordination between the agents. This apparent discrepancy is due to a phenomenon coined the team uncertainty penalty (Taylor et al, 2011), in which agents with few neighbours actually achieve higher performance with lower levels of coordination (e.g. K1), while agents with many neighbours can achieve higher performance with higher levels of coordination (e.g.…”

“…However, the DCOP framework requires that the reward of every action combination be known a priori, making it difficult to handle non-stationary traffic distributions. In contrast, our previous work has extended the DCOP framework to the distributed coordination of exploration and exploitation (DCEE) (Taylor, Jain, Tandon, Yokoo, & Tambe, 2011) framework. In order to address the importance of dynamic and unknown rewards, DCEE algorithms take a multi-agent approach towards balancing exploiting known good configurations with exploration of novel action combinations to attempt to find better rewards.…”

A Tutorial on Optimization for Multi-Agent Systems

“…Hence the uncertainty must be somehow captured into the DCOP model and dealt with in the solution technique. There are a number of approaches that tackle these issues, in particular, we refer the interest reader to [66] for approaches that model uncertainty of the reward and attempts to nd optimal solution considering such uncertainty, and to [67,68] for approaches that aim at learning unknown rewards of agents' joint actions.…”