The Frontiers of Deep Reinforcement Learning for Resource Management in Future Wireless HetNets: Techniques, Challenges, and Research Directions

Abstract

Next generation wireless networks are expected to be extremely complex due to their massive heterogeneity in terms of the types of network architectures they incorporate, the types and numbers of smart IoT devices they serve, and the types of emerging applications they support. In such large-scale and heterogeneous networks (HetNets), radio resource allocation and management (RRAM) becomes one of the major challenges encountered during system design and deployment. In this context, emerging Deep Reinforcement Learning (DRL) techniques are expected to be one of the main enabling technologies to address the RRAM in future wireless HetNets. In this paper, we conduct a systematic in-depth, and comprehensive survey of the applications of DRL techniques in RRAM for next generation wireless networks. Towards this, we first overview the existing traditional RRAM methods and identify their limitations that motivate the use of DRL techniques in RRAM. Then, we provide a comprehensive review of the most widely used DRL algorithms to address RRAM problems, including the value-and policy-based algorithms. The advantages, limitations, and use-cases for each algorithm are provided. We then conduct a comprehensive and in-depth literature review and classify existing related works based on both the radio resources they are addressing and the type of wireless networks they are investigating. To this end, we carefully identify the types of DRL algorithms utilized in each related work, the elements of these algorithms, and the main findings of each related work. Finally, we highlight important open challenges and provide insights into several future research directions in the context of DRL-based RRAM. This survey is intentionally designed to guide and stimulate more research endeavors towards building efficient and fine-grained DRL-based RRAM schemes for future wireless networks.

Deep Reinforcement Learning for Internet of Drones Networks: Issues and Research Directions

“…There are several survey papers on the applications of DRL techniques for various issues and applications in wireless networks and for resource allocation and management in 5G and beyond wireless networks [1], [7]. Although there are papers that discuss the applications of machine learning, deep learning, and DRL for IoD, the contribution and values added by this paper compared to these paper is summarized in Table I.…”

“…Experts also expect that existing conventional techniques, such as optimization-and game-theory-based, will encounter serious inadequacies when addressing the above challenges due to the high complexity and dynamicity of the IoD environments [7], [8]. Therefore, emerging machine learning approaches, such as deep reinforcement learning (DRL) techniques, have been proposed lately to be utilized instead.…”

“…In addition, in complicated and large-scale IoD networks, the computational complexity of conventional algorithms becomes extremely challenging to manage. Hence, DRL has been presented lately to overcome such challenges in highly dynamic environments with sophisticated state spaces [7]. DRL utilizes Deep Neural Networks (DNNs) to offer efficient implementation for various decisionmaking tasks with local knowledge of IoD environments [9].…”

Data-Driven Participant Selection and Bandwidth Allocation for Heterogeneous Federated Edge Learning

“…Throughout this paper, we only consider the uplink latency, and we assume the downlink latency is negligible due to the powerful capabilities and the sufficient bandwidth of the edge server. We note that the computation time is proportional to the collected data and CPU frequency as in (9). Also, the uplink latencies depend on the channel state, transmission power, and model size as in (11).…”

“…the model is continuously updated when new samples arrive. Such centralized approaches are becoming increasingly costly since offloading high dimensional data from end devices to the edge server in intelligent systems is often infeasible due to limited wireless resources, latency, and privacy concerns [8], [9]. Therefore, the data generated at the edge devices needs to be stored and processed locally.…”