Reinforcement Learning for Real-Time Optimization in NB-IoT Networks

Abstract

NarrowBand-Internet of Things (NB-IoT) is an emerging cellular-based technology that offers a range of flexible configurations for massive IoT radio access from groups of devices with heterogeneous requirements. A configuration specifies the amount of radio resource allocated to each group of devices for random access and for data transmission. Assuming no knowledge of the traffic statistics, there exists an important challenge in "how to determine the configuration that maximizes the long-term average number of served IoT devices at each Transmission Time Interval (TTI) in an online fashion". Given the complexity of searching for optimal configuration, we first develop real-time configuration selection based on the tabular Q-learning (tabular-Q), the Linear Approximation based Q-learning (LA-Q), and the Deep Neural Network based Q-learning (DQN) in the single-parameter single-group scenario. Our results show that the proposed reinforcement learning based approaches considerably outperform the conventional heuristic approaches based on load estimation (LE-URC) in terms of the number of served IoT devices. This result also indicates that LA-Q and DQN can be good alternatives for tabular-Q to achieve almost the same performance with much less training time. We further advance LA-Q and DQN via Actions Aggregation (AA-LA-Q and AA-DQN) and via Cooperative Multi-Agent learning (CMA-DQN) for the multi-parameter multi-group scenario, thereby solve the problem that Q-learning agents do not converge in high-dimensional configurations. In this scenario, the superiority of the proposed Q-learning approaches over the conventional LE-URC approach significantly improves with the increase of configuration dimensions, and the CMA-DQN approach outperforms the other approaches in both throughput and training efficiency. Partnership Project (3GPP) partners have standardized a new radio access technology, namely NarrowBand-IoT (NB-IoT) [1]. NB-IoT is expected to provide reliable wireless access for IoT devices with various

Explainable Artificial Intelligence for 6G: Improving Trust between Human and Machine

“…Neural networks (NN) with multiple layers cannot explain the essential features that influence actions, nor the impact of data bias on the uncertainty of outputs. Beyond supervised learning for PHY layer signal detection, DNN is especially opaque when coupled with reinforcement learning (RL) [3], where the Markov Decision Process (MDP) is integrated with hidden layer dynamics. As such, there is the need to develop explainable algorithms that can quantify uncertainty, especially mapping data inputs, algorithm design, to the projected wireless key performance indicators (KPI).…”

Scheduling and Decoding of Downlink Control Channel in 3GPP Narrowband-IoT

“…In [6], [7], [8], a detailed explanation of NB-IoT design and physical layer procedures has been presented. In [9], a reinforcement learning-based framework to configure resources optimally for uplink in NB-IoT has been presented. In [10], authors have proposed an uplink link-adaptation scheme for the IoT devices in an NB-IoT network.…”

“…(i) With α = 0, the proposed scheduler in (9) becomes an objective function that tries to schedule more number of devices in a search space. Thus, the number of devices scheduled will always be greater than or equal to that of the other schedulers.…”

Special Issue on Artificial Intelligence and Machine Learning for Networking and Communications

“…Jiang et al address the problem of resource allocation in NB-IoT in their paper entitled "Reinforcement learning for real-time optimization in NB-IoT networks". They provide several proposals using reinforcement learning to maximize the number of served IoT devices in dynamic environments [14]. In the paper, the authors show that their approach requires less training time while the improvement of their algorithm using cooperative multi-agent learning outperforms other methods and converges to efficient solutions even at large scale.…”