D&lt;inf&gt;++&lt;/inf&gt;: Structural credit assignment in tightly coupled multiagent domains

Rahmattalabi, Aida; Chung, Jen Jen; Colby, Mitchell; Tumer, Kagan

doi:10.1109/iros.2016.7759651

Cited by 25 publications

(14 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Notably, MERL is able to learn on coupling greater than n = 6 where methods without explicit reward shaping have been shown to fail entirely [Rahmattalabi et al, 2016]. This is consistent with the performances of our baselines as none of them use explicit domain-specific reward shaping.…”

Section: Resultssupporting

confidence: 83%

Evolutionary Reinforcement Learning for Sample-Efficient Multiagent Coordination

Khadka,

Majumdar,

Miret

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

A key challenge for Multiagent RL (Reinforcement Learning) is the design of agent-specific, local rewards that are aligned with sparse global objectives. In this paper, we introduce MERL (Multiagent Evolutionary RL), a hybrid algorithm that does not require an explicit alignment between local and global objectives. MERL uses fast, policy-gradient based learning for each agent by utilizing their dense local rewards. Concurrently, an evolutionary algorithm is used to recruit agents into a team by directly optimizing the sparser global objective. We explore problems that require coupling (a minimum number of agents required to coordinate for success), where the degree of coupling is not known to the agents. We demonstrate that MERL's integrated approach is more sample-efficient and retains performance better with increasing coupling orders compared to MADDPG, the state-of-the-art policy-gradient algorithm for multiagent coordination.

show abstract

Section: Resultssupporting

confidence: 83%

Evolutionary Reinforcement Learning for Sample-Efficient Multiagent Coordination

Khadka,

Majumdar,

Miret

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…In practice, optimizing with respect to f r rather than g improves the performance since f r is more sensitive to robot r ’s plan and the variance of f r is less affected by the uncertainty of the other robots’ plans (Wolpert et al, 2013). We chose this local utility function since it is generally applicable, although further performance improvements could be achieved with problem-specific heuristics (Rahmattalabi et al, 2016). We note that this formulation assumes that all robots know the global utility function g .…”

Section: Dec-mctsmentioning

confidence: 99%

Dec-MCTS: Decentralized planning for multi-robot active perception

Best

Cliff

Patten

et al. 2018

The International Journal of Robotics Research

153

138

View full text Add to dashboard Cite

We propose a decentralized variant of Monte Carlo tree search (MCTS) that is suitable for a variety of tasks in multi-robot active perception. Our algorithm allows each robot to optimize its own actions by maintaining a probability distribution over plans in the joint-action space. Robots periodically communicate a compressed form of their search trees, which are used to update the joint distribution using a distributed optimization approach inspired by variational methods. Our method admits any objective function defined over robot action sequences, assumes intermittent communication, is anytime, and is suitable for online replanning. Our algorithm features a new MCTS tree expansion policy that is designed for our planning scenario. We extend the theoretical analysis of standard MCTS to provide guarantees for convergence rates to the optimal payoff sequence. We evaluate the performance of our method for generalized team orienteering and online active object recognition using real data, and show that it compares favorably to centralized MCTS even with severely degraded communication. These examples demonstrate the suitability of our algorithm for real-world active perception with multiple robots.

show abstract

“…Although the agents counld use the global utility o directly, optimising with respect to f i instead results in faster convergence since f i is less affected by the unknown plans of teammates, aka. World Utility (Rahmattalabi et al 2016).…”

Section: Phase 1: Local Sub-goal Tree Searchmentioning

confidence: 99%

Dec-SGTS: Decentralized Sub-Goal Tree Search for Multi-Agent Coordination

Cai

Yang

et al. 2021

AAAI

View full text Add to dashboard Cite

Multi-agent coordination tends to benefit from efficient communication, where cooperation often happens based on exchanging information about what the agents intend to do, i.e. intention sharing. It becomes a key problem to model the intention by some proper abstraction. Currently, it is either too coarse such as final goals or too fined as primitive steps, which is inefficient due to the lack of modularity and semantics. In this paper, we design a novel multi-agent coordination protocol based on subgoal intentions, defined as the probability distribution over feasible subgoal sequences. The subgoal intentions encode macro-action behaviors with modularity so as to facilitate joint decision making at higher abstraction. Built over the proposed protocol, we present Dec-SGTS (Decentralized Sub-Goal Tree Search) to solve decentralized online multi-agent planning hierarchically and efficiently. Each agent runs Dec-SGTS asynchronously by iteratively performing three phases including local sub-goal tree search, local subgoal intention update and global subgoal intention sharing. We conduct the experiments on courier dispatching problem, and the results show that Dec-SGTS achieves much better reward while enjoying a significant reduction of planning time and communication cost compared with Dec-MCTS (Decentralized Monte Carlo Tree Search).

show abstract

D<inf>++</inf>: Structural credit assignment in tightly coupled multiagent domains

Cited by 25 publications

References 12 publications

Evolutionary Reinforcement Learning for Sample-Efficient Multiagent Coordination

Evolutionary Reinforcement Learning for Sample-Efficient Multiagent Coordination

Dec-MCTS: Decentralized planning for multi-robot active perception

Dec-SGTS: Decentralized Sub-Goal Tree Search for Multi-Agent Coordination

Contact Info

Product

Resources

About