Fast Q-Learning for Improved Finite Length Performance of Irregular Repetition Slotted ALOHA

Abstract

In this paper, we study the problem of designing adaptive Medium Access Control (MAC) solutions for wireless sensor networks (WSNs) under the Irregular Repetition Slotted ALOHA (IRSA) protocol. In particular, we optimize the degree distribution employed by IRSA for finite frame sizes. Motivated by characteristics of WSNs, such as the restricted computational resources and partial observability, we model the design of IRSA as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP). We have theoretically analyzed our solution in terms of optimality of the learned IRSA design and derived guarantees for finding near-optimal policies. These guarantees are generic and can be applied in resource allocation problems that exhibit the waterfall effect, which in our setting manifests itself as a severe degradation in the overall throughput of the network above a particular channel load. Furthermore, we combat the inherent non-stationarity of the learning environment in WSNs by advancing classical Q-learning through the use of virtual experience (VE), a technique that enables the update of multiple state-action pairs per learning iteration and, thus, accelerates convergence. Our simulations confirm the superiority of our learning-based MAC solution compared to traditional IRSA and provide insights into the effect of WSN characteristics on the quality of learned policies.

Adaptive Wireless Network Management with Multi-Agent Reinforcement Learning

“…Nisioti et al [ 21 ] proposed a MAC design for wireless sensor networks utilising the irregular repetition slotted ALOHA protocol. Like ours, their approach uses multiple RL agents, deployed on the nodes, in a decentralised manner.…”

An Artificial Intelligence-Based Quorum System for the Improvement of the Lifespan of Sensor Networks

“…This approach reduces energy and latency, and compared with existing protocols improves its network lifetime. A decentralized adaptive duty cycle is framed for wireless sensor networks for slotted ALOHA [27] using the Markov decision estimation procedure. The work employed Q-learning concepts for providing a solution for optimization, which increases the throughput of the packet in the channel.…”

Random Interleaving Multiplexing Based IRSA Random Access System for Satellite-Enabled Internet-of-Things

“…To further improve the throughput of RA schemes, some new schemes are proposed to resolve multiple packet collisions issue, such as those in literature [10]- [15]. In [10], the authors propose an IRSA with energy harvesting nodes RA protocol.…”

“…Although the throughput can be improved, the scheme is difficult to deploy for devices without polarized antennas. The authors in [15] proposed the Q-learning based IRSA scheme, however it can not be applied into the satellite-enabled IoTs because of the delay problem caused by online learning. From the perspective of code-domain non-orthogonal multiple access (NOMA), we utilized the interleaver randomly selected to distinguish different users and proposed a random interleaving multiplexing based CRDSA protocol in [16].…”