Analyzing and visualizing multiagent rewards in dynamic and stochastic domains

Abstract

The ability to analyze the effectiveness of agent reward structures is critical to the successful design of multiagent learning algorithms. Though final system performance is the best indicator of the suitability of a given reward structure, it is often preferable to analyze the reward properties that lead to good system behavior (i.e., properties promoting coordination among the agents and providing agents with strong signal to noise ratios). This step is particularly helpful in continuous, dynamic, stochastic domains ill-suited to simple table backup schemes commonly used in TD(λ)/Q-learning where the effectiveness of the reward structure is difficult to distinguish from the effectiveness of the chosen learning algorithm. In this paper, we present a new reward evaluation method that provides a visualization of the tradeoff between the level of coordination among the agents and the difficulty of the learning problem each agent faces. This method is independent of the learning algorithm and is only a function of the problem domain and the agents' reward structure. We use this reward property visualization method to determine an effective reward without performing extensive simulations. We then test this method in both a static and a dynamic multi-rover learning domain where the agents have continuous state spaces and take noisy actions (e.g., the agents' movement decisions are not always carried out properly). Our results show that in the more difficult dynamic domain, the reward efficiency visualization method provides a two order of magnitude speedup in selecting good rewards, compared to running a full simulation. In addition, this method facilitates the design and analysis of new rewards tailored to the observational limitations of the domain, providing rewards that combine the best properties of traditional rewards.

Distributed sensor coordination for advanced energy systems

“…Thus, an agent acting to increase the Difference Reward will also act to increase the global reward. This property is termed factoredness [5]. Further, because the Difference Reward only depends on the actions of agent i, noise from other agents is reduced in the feedback given by D i .…”

Coevolution of heterogeneous multi-robot teams

“…The weights of the neural network are adjusted through an evolutionary search algorithm [3,2] for ranking and subsequently locating successful networks within a population [12,3]. The algorithm maintains a population of ten networks, utilizes mutation to modify individuals, and ranks them based on a performance metric specific to the domain.…”

“…• The difference evaluation reflects the impact a robot has on the full system [3,2]. By removing the value of the system evaluation where robot i is inactive, the difference evaluation computes the value added by the observations of robot i alone.…”

Learning From Actions Not Taken in Multiagent Systems

“…As a consequence, in this work, we use the difference reward as a starting point for the reward an agent receives after each step. Earlier work has shown that the difference reward significantly outperforms both agents receiving a purely local reward and all agents receiving the same system reward [3,2,33,32,36]. The difference reward is given by:…”

“…In these cases, the learning needs of the agents are modified to account for their presence in a larger system [2,11,13,22,35,37]. However, though these methods have yielded tremendous advances in multiagent learning, they are principally based on an agent trying an action, receiving an evaluation of that action, and updating its own estimate on the "value" of taking that action in that state.…”