The Emergence of Wireless MAC Protocols with Multi-Agent Reinforcement Learning

Learning Generalized Wireless MAC Communication Protocols via Abstraction

“…We underline that we set R t = −1 when no condition is true to minimize the number of time slots. The values assigned to the reward R t follow from [4]. However, unlike that work, where the agents are positively rewarded if condition 3) is true, we also give a positive reward if condition 2) is true, since it allows the agent to transmit, in the subsequent time steps the next packet in the buffer.…”

“…To learn it, we adopt the CTDE paradigm. In general, several MARL techniques can be used, ranging from off-policy learning frameworks, such as MADDPG [4], value-based approaches (e.g., tabular Qlearning [5]), to on-policy algorithms such as MAPPO [9]. Among them, in this work we adopt MAPPO, since its onpolicy nature is well-suited to the task of learning new MAC protocols.…”

“…In particular, multiagent reinforcement learning (MARL) [2] methods enable agents to learn an optimal policy by interacting with nonstationary environments. Recent advances in deep RL and learning-to-communicate techniques (e.g., Differentiable Inter-Agent Learning (DIAL), and Reinforced Inter-Agent Learning (RIAL) [3]) have led to the emergence of protocol learning for the physical (PHY) and media access control (MAC) layers [4]- [7]. Among them, to the best of our knowledge, the only works that assess the problem of MAC protocol learning in both control and data planes are [4], [5].…”

“…Recent advances in deep RL and learning-to-communicate techniques (e.g., Differentiable Inter-Agent Learning (DIAL), and Reinforced Inter-Agent Learning (RIAL) [3]) have led to the emergence of protocol learning for the physical (PHY) and media access control (MAC) layers [4]- [7]. Among them, to the best of our knowledge, the only works that assess the problem of MAC protocol learning in both control and data planes are [4], [5]. Therein, user equipments (UEs) are cast as agents that learn from their partial observation of the global state how to deliver MAC protocol data units (PDUs) to the base station (BS) throughout the radio channel.…”

“…To generate optimal policies, the centralized training and decentralized execution (CTDE) paradigm is adopted, where agents are trained offline using centralized information but execute in a decentralized manner. Specifically, in [4], both the BS and the UEs are cast as RL agents, and the multi-agent deep deterministic policy gradient (MADDPG) algorithm is adopted, that is, a commonly used CTDE-based actor-critic method. In [5], the BS is modeled as an expert agent adopting a predefined protocol, while the UEs are RL agents trained to learn a shared channel-access policy following the target signaling policy set by the BS.…”