Adaptive Contention Window Design Using Deep Q-Learning

Abstract

We study the problem of adaptive contention window (CW) design for random-access wireless networks. More precisely, our goal is to design an intelligent node that can dynamically adapt its minimum CW (MCW) parameter to maximize a network-level utility knowing neither the MCWs of other nodes nor how these change over time.To achieve this goal, we adopt a reinforcement learning (RL) framework where we circumvent the lack of system knowledge with local channel observations and we reward actions that lead to high utilities. To efficiently learn these preferred actions, we follow a deep Q-learning approach, where the Q-value function is parametrized using a multi-layer perceptron. In particular, we implement a rainbow agent, which incorporates several empirical improvements over the basic deep Q-network. Numerical experiments based on the NS3 simulator reveal that the proposed RL agent performs close to optimal and markedly improves upon existing learning and non-learning based alternatives.

Wi-Fi Meets ML: A Survey on Improving IEEE 802.11 Performance with Machine Learning

“…As summarized in Fig. 3, SL [38], [40], RL [36], [37], [45], deep reinforcement learning (DRL) [39], [46], [48], and federated learning (FL) [47], [48] models are applied to the IEEE 802.11 standard [37], [48] and its amendments, most importantly 802.11ac [38], 802.11e [36], [43], 802.11n [40], and 802.11ax [39], [46]. We provide a summary of the major findings next.…”

“…Determining proper CW values to maximize throughput by reducing both collisions and idle time periods is the focus of multiple research studies, where SL and RL models are typically applied. Loss functions and rewards are addressed in the form of reduced collisions [39], [45], increased difference between successful and collided frames [36], improved channel utilization [38], increased successful channel access attempts [40], [48], throughput [46], network utility [80], and a combination of throughput, energy, and collisions [37]. As summarized in Fig.…”

Applying Deep Reinforcement Learning to Improve Throughput and Reduce Collision Rate in IEEE 802.11 Networks

“…Compared to simple Q-learning, DQN is an additional DNN to enable more effective reward extrapolation with yet unseen states. DQN algorithm is also used for CW optimization in more complex network scenarios to maximize a network-level utility used in references [24], [25]. Kumar et and Reduce Collision Rate in IEEE 802.11 Networks al.…”

“…Kumar et and Reduce Collision Rate in IEEE 802.11 Networks al. [24] designed an intelligent node that can dynamically adapt its minimum CW (MCW) parameter to maximize a network-level utility without knowing the MCW values in other nodes. In another study [25], the authors proposed a self-adaptive MAC layer algorithm employing DQN with a novel contention information-based state representation to improve the performance of the V2V safety packet broadcast.…”

“…In addition, the RL and DRL algorithms can now be used to study CW optimization because of the high computing capabilities in modern network devices. Some recent works [16], [21][22][23][24][25] discuss the CW optimization through the effectiveness of Q-learning and deep Q-learning network (DQN) algorithms, thus describing the problem of optimizing the CW value in mobile ad-hoc networks (MANETs), VANETs, and for both LTE-LAA and Wi-Fi networks.…”