Multiagent Deep Reinforcement Learning: Challenges and Directions Towards Human-Like Approaches

Abstract

This paper surveys the field of multiagent deep reinforcement learning. The combination of deep neural networks with reinforcement learning has gained increased traction in recent years and is slowly shifting the focus from single-agent to multiagent environments. Dealing with multiple agents is inherently more complex as (a) the future rewards depend on multiple players' joint actions and (b) the computational complexity of functions increases. We present the most common multiagent problem representations and their main challenges, and identify five research areas that address one or more of these challenges: centralised training and decentralised execution, opponent modelling, communication, efficient coordination, and reward shaping. We find that many computational studies rely on unrealistic assumptions or are not generalisable to other settings; they struggle to overcome the curse of dimensionality or nonstationarity. Approaches from psychology and sociology capture promising relevant behaviours such as communication and coordination. We suggest that, for multiagent reinforcement learning to be successful, future research addresses these challenges with an interdisciplinary approach to open up new possibilities for more human-oriented solutions in multiagent reinforcement learning.

Deep Reinforcement Learning Based Beamforming Codebook Design for RIS-aided mmWave Systems

“…This section develops a solution to the joint beamforming and codebook design by leveraging powerful exploration capability of MA-DRL to find a near-optimal solution over the huge search space mentioned above. The MA-DRL differs from its single agent counterpart in cooperating and acting jointly to achieve a common ultimate reward [15]. The MA-DRL is especially suitable for complex problems that can be decomposable into sub-problems, each of which is handled by a single DRL agents.…”

Social Neuro AI: Social Interaction as the “Dark Matter” of AI

“…At each timestep t, each agent is attempting to maximize its own reward by learning a policy that optimizes the total expected discounted future reward. We refer the reader to high-quality reviews that have been written on MARL (Hernandez-Leal et al, 2019;Nguyen et al, 2020;Wong et al, 2021). Here, we highlight that, among others, low sample efficiency is one of the greatest challenges for MARL, as millions of interactions with the environment are usually needed for agents to learn.…”

Evolving Modular Soft Robots without Explicit Inter-Module Communication using Local Self-Attention

“…We consider the distributed controller proposed in [47], consisting in a number of fully-connected, feed-forward Artificial Neural Networks (ANNs), one for every voxel. In particular, we adopt the "homogeneous" variant presented in [49], where the ANNs share the same parameters: Medvet et al [49] proved that such homogeneous representation is comparable to one where parameters are different for every ANN, with the additional benefit of a more compact search space, similarly to what happens in most multi-agent reinforcement learning systems [76]. Moreover, parameter sharing makes the controller agnostic with respect to the morphology, putting ourselves on a vantage point to test generalization to unseen morphologies.…”

“…Modularity is also ubiquitous in the field of artificial intelligence: it appears in Graph Neural Networks (GNNs) [61], Cellular Automata (CA) [51], and multi-agent systems [76]. In evolutionary robotics, Voxel-based Soft Robots (VSRs) [29] are simulated aggregations of mechanically identical elastic blocks: as such, they have emerged as as a relevant formalism to model state-of-the-art robotic systems, e.g., soft robotics [59].…”