Multiagent Cooperation and Competition with Deep Reinforcement Learning

Tampuu, Ardi; Matiisen, Tambet; Kodelja, Dorian; Kuzovkin, Ilya; Korjus, Kristjan; Aru, Juhan; Aru, Jaan; Vicente, Raúl

doi:10.48550/arxiv.1511.08779

Cited by 14 publications

(24 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To empirically support our theoretical results on EMGs, this section is devoted to introduce two simple decentralized extensions of singleagent Deep Reinforcement Learning (DRL) algorithms designed to work with LT L specifications. The first approach is based on extending with temporal logic specifications a popular baseline in MARL called I-DQN [30], while the second is a multi-agent extension of LPOPL that we referred in Sec 1. The extended algorithms described below are employed in the experiments presented in Sec.…”

Section: Deep Marl With Co-safe Lt L Goalsmentioning

confidence: 99%

“…This allows Independent Q-learning [32] to train multiple agents in a decentralized fashion. Here we consider a deep learning variant of this algorithm (see, e.g., [30]), where each agent is trained with an independent DQN. However, in our case, we adopt a decentralized version of an algorithm that uses LT L specifications and LT L progression instead of classical reward functions (see, e.g., [20]).…”

Section: I-dqn With Co-safe Lt L Goalsmentioning

confidence: 99%

“…Despite the growing success of MARL, also due to the adoption of neural networks in Deep (RL) [30], there is still little work on combining these multi-agent learning techniques with formal methods. This is the long-term challenge that we here start to address.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Extended Markov Games to Learn Multiple Tasks in Multi-Agent Reinforcement Learning

León,

Belardinelli

2020

Preprint

View full text Add to dashboard Cite

The combination of Formal Methods with Reinforcement Learning (RL) has recently attracted interest as a way for singleagent RL to learn multiple-task specifications. In this paper we extend this convergence to multi-agent settings and formally define Extended Markov Games as a general mathematical model that allows multiple RL agents to concurrently learn various non-Markovian specifications. To introduce this new model we provide formal definitions and proofs as well as empirical tests of RL algorithms running on this framework. Specifically, we use our model to train two different logic-based multi-agent RL algorithms to solve diverse settings of non-Markovian co-safe LT L specifications.

show abstract

Section: Deep Marl With Co-safe Lt L Goalsmentioning

confidence: 99%

Section: I-dqn With Co-safe Lt L Goalsmentioning

confidence: 99%

See 1 more Smart Citation

Extended Markov Games to Learn Multiple Tasks in Multi-Agent Reinforcement Learning

León,

Belardinelli

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Cooperative Cooperative CAD environments help in developing agent algorithms that can learn near-globally optimal policies for all the driving agents that act as a cooperative unit. Such environments help in developing agents that learn to communicate [9] and benefit from learning to cooperate [25]. This type of environments will enable development of efficient fleet of vehicles that cooperate and communicate with each other to reduce congestion, eliminate collisions and optimized traffic flows.…”

Section: Nature Of Tasksmentioning

confidence: 99%

“…Independent DQN [25] extends DQN to cooperative, fully-observable Multi-Agent setting, applied to a two-player pong environment, in which all agents independently learn and update their own Q-function Q i (s, a i ; θ i ).…”

Section: A Appendixamentioning

confidence: 99%

Multi-Agent Connected Autonomous Driving using Deep Reinforcement Learning

Palanisamy¹

2019

Preprint

View full text Add to dashboard Cite

The capability to learn and adapt to changes in the driving environment is crucial for developing autonomous driving systems that are scalable beyond geo-fenced operational design domains. Deep Reinforcement Learning (RL) provides a promising and scalable framework for developing adaptive learning based solutions. Deep RL methods usually model the problem as a (Partially Observable) Markov Decision Process in which an agent acts in a stationary environment to learn an optimal behavior policy. However, driving involves complex interaction between multiple, intelligent (artificial or human) agents in a highly non-stationary environment. In this paper, we propose the use of Partially Observable Markov Games(POSG) for formulating the connected autonomous driving problems with realistic assumptions. We provide a taxonomy of multi-agent learning environments based on the nature of tasks, nature of agents and the nature of the environment to help in categorizing various autonomous driving problems that can be addressed under the proposed formulation. As our main contributions, we provide MACAD-Gym, a Multi-Agent Connected, Autonomous Driving agent learning platform for furthering research in this direction. Our MACAD-Gym platform provides an extensible set of Connected Autonomous Driving (CAD) simulation environments that enable the research and development of Deep RL-based integrated sensing, perception, planning and control algorithms for CAD systems with unlimited operational design domain under realistic, multi-agent settings. We also share the MACAD-Agents that were trained successfully using the MACAD-Gym platform to learn control policies for multiple vehicle agents in a partially observable, stop-sign controlled, 3-way urban intersection environment with raw (camera) sensor observations.

show abstract

Deep Q-Learning-Based Dynamic Management of a Robotic Cluster

Gautier

Laurent

Diguet³

2023

IEEE Trans. Automat. Sci. Eng.

View full text Add to dashboard Cite

The ever-increasing demands for autonomy and precision have led to the development of heavily computational multi-robot system (MRS). However, numerous missions exclude the use of robotic cloud. Another solution is to use the robotic cluster to locally distribute the computational load. This complex distribution requires adaptability to come up with a dynamic and uncertain environment. Classical approaches are too limited to solve this problem, but recent advances in reinforcement learning and deep learning offer new opportunities. In this paper we propose a new Deep Q-Network (DQN) based approaches where the MRS learns to distribute tasks directly from experience. Since the problem complexity leads to a curse of dimensionality, we use two specific methods, a new branching architecture, called Branching Dueling Q-Network (BDQ), and our own optimized multi-agent solution and we compare them with classical Marketbased approaches as well as with non-distributed and purely local solutions. Our study shows the relevancy of learning-based methods for task mapping and also highlight the BDQ architecture capacity to solve high dimensional state space problems.Note to Practitioners-A lot of applications in industry like area exploration and monitoring can be efficiently delegated to a group of small-size robots or autonomous vehicles with advantages like reliability and cost in respect of single-robot solutions. But autonomy requires high and increasing compute-intensive tasks such as computer-vision. On the other hand small robots have energy constraints, limited embedded computing capacities and usually restricted and/or unreliable communications that limit the use of cloud resources. An alternative solution to cope with this problem consists in sharing the computing resources of the group of robots. Previous work was a proof of concept limited to the parallelisation of a single specific task. In this paper we formalize a general method that allows the group of robots to learn on the field how to efficiently distribute tasks in order to optimize the execution time of a mission under energy constraint. We demonstrate the relevancy of our solution over market-based and non-distributed approaches by means of intensive simulations. This successful study is a necessary first step towards distribution and parallelisation of computation tasks over a robotic cluster. The next steps, not tested yet, will address hardware in the loop simulation and finally a real-life mission with a group of robots.

show abstract

Multiagent Cooperation and Competition with Deep Reinforcement Learning

Cited by 14 publications

References 0 publications

Extended Markov Games to Learn Multiple Tasks in Multi-Agent Reinforcement Learning

Extended Markov Games to Learn Multiple Tasks in Multi-Agent Reinforcement Learning

Multi-Agent Connected Autonomous Driving using Deep Reinforcement Learning

Deep Q-Learning-Based Dynamic Management of a Robotic Cluster

Contact Info

Product

Resources

About