Deep Q-Learning for Two-Hop Communications of Drone Base Stations

Fotouhi, Azade; Ding, Ming; Hassan, Mahbub

doi:10.3390/s21061960

Cited by 12 publications

(6 citation statements)

References 31 publications

(30 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…IAB has been justified for usage over 5G infrastructure by the 3GPP [114] and is deemed as useful in enhancing capacity, coverage, as well as connectivity. However, additional challenges are imposed on the UAV that needs to guarantee stable backhaul and access links [115]. Cao et al in [116] proposed a UEdriven deep reinforcement learning (DRL) based scheme, in which a centralized agent deployed at the backhaul side of NT-BSs is responsible for training the parameter of a deep Qnetwork (DQN), and each UE is able to access a proper NT-BS intelligently to enhance the long-term system throughput and avoid frequent handovers among NT-BSs.…”

Section: B Integrated Access and Backhaulmentioning

confidence: 99%

Reinforcement Learning in the Sky: A Survey on Enabling Intelligence in NTN-Based Communications

et al. 2023

View full text Add to dashboard Cite

Non terrestrial networks (NTN) involving 'in the sky' objects such as low-earth orbit satellites, high altitude platform systems (HAPs) and Unmanned Aerial Vehicles (UAVs) are expected to be integral components of next generation cellular systems. With the deployment of 5G services and beyond, NTNs are leveraged to assist as aerial base stations in providing ubiquitous network connectivity and service to ground users or be deployed as aerial users connected to the cellular network. NTN-aided wireless communication offers multiple benefits such as mobility, flexibility, resistance to ground physical attacks and wide coverage. However, due to their limited resources and the current design of terrestrial cellular systems that do not account for aerial users, and other restrictions such as service requirements, limited available power and storage resources on high-throughput satellites, resource allocation, location of the high altitude platform base station and the flight trajectory of the UAVs need to be intelligently controlled to satisfy various objectives both from an aerial base station and overall network perspectives. To achieve this, many works have explored Reinforcement Learning (RL) techniques to allow aerial platforms in non-terrestrial networks to learn from past observations and achieve some optimal control policy. In this paper and differently from prior surveys, we contribute a comprehensive review of the control objectives required by non-terrestrial platforms that have been solved using RL formulations. We provide an up-to-date overview of the latest applications of RL techniques for different NTN-aided wireless communication aspects. The survey focuses on Markov Decision Process (MDP) formulations in terms of states, actions, and rewards. We synthesize a taxonomy from the surveyed literature and provide a comprehensive representation of the current usages of RL in NTN-aided wireless communications. A qualitative analysis of the level of realism achieved in the works presented in the literature is provided based on several factors that pertain to the simulation environment, station deployment setting, wireless channel assumption, and energy considerations. We also curate a list of challenges that remain to be considered by the research community in order to achieve more efficient deployments and close the simulation-to-reality gap.

show abstract

Section: B Integrated Access and Backhaulmentioning

confidence: 99%

Reinforcement Learning in the Sky: A Survey on Enabling Intelligence in NTN-Based Communications

et al. 2023

View full text Add to dashboard Cite

show abstract

“…Returning to Equation ( 9), x v is given by Equation ( 14), and the 3-dB beamwidth in the elevation plane θ 3 is calculated using Equation (15), where G 0 represents the maximum gain in the azimuth plane. In addition, x k is calculated using Equation ( 16), where k v is an elevation pattern adjustment factor based on the leaked power.…”

Section: Interfered System Modelmentioning

confidence: 99%

Deep Q-Learning-Based Transmission Power Control of a High Altitude Platform Station with Spectrum Sharing

Yang

Choi

et al. 2022

Sensors

View full text Add to dashboard Cite

A High Altitude Platform Station (HAPS) can facilitate high-speed data communication over wide areas using high-power line-of-sight communication; however, it can significantly interfere with existing systems. Given spectrum sharing with existing systems, the HAPS transmission power must be adjusted to satisfy the interference requirement for incumbent protection. However, excessive transmission power reduction can lead to severe degradation of the HAPS coverage. To solve this problem, we propose a multi-agent Deep Q-learning (DQL)-based transmission power control algorithm to minimize the outage probability of the HAPS downlink while satisfying the interference requirement of an interfered system. In addition, a double DQL (DDQL) is developed to prevent the potential risk of action-value overestimation from the DQL. With a proper state, reward, and training process, all agents cooperatively learn a power control policy for achieving a near-optimal solution. The proposed DQL power control algorithm performs equal or close to the optimal exhaustive search algorithm for varying positions of the interfered system. The proposed DQL and DDQL power control yields the same performance, which indicates that the actional value overestimation does not adversely affect the quality of the learned policy.

show abstract

“…In this regard, local dithering techniques like ϵgreedy algorithm is commonly used to allow large exploration (with probability ϵ) at the beginning of the training phase and later increases exploitation with increasing number of training episodes. In recent studies, [38]- [40] DQN is often used with the aforementioned tricks and trades (experienced replay, target network with ϵ-greedy algorithm). Collectively, we refer to them as vanilla DQN for the rest of the article.…”

Section: A Reinforcement Learning and Vanilla Dqnmentioning

confidence: 99%

An Efficient Computing of Correlated Equilibrium for Cooperative $Q$-Learning-Based Multi-Robot Planning

Sadhu

Konar

2019

IEEE Trans. Syst. Man Cybern, Syst.

View full text Add to dashboard Cite

Recent advancements in deep reinforcement learning (DRL) techniques have sparked its multifaceted applications in the automation sector. Managing complex decision-making problems with DRL encourages its use in the nuclear industry for tasks such as optimizing radiation exposure to the personnel during normal operating conditions and potential accidental scenarios. However, the lack of efficient reward function and effective exploration strategy thwarted its implementation in the development of radiation-aware autonomous unmanned aerial vehicle (UAV) for achieving maximum radiation protection. Here, in this article, we address these intriguing issues and introduce a deep Q-learning based architecture (RadDQN) that operates on a radiation-aware reward function to provide time-efficient minimum radiation-exposure pathway in a radiation zone. We propose a set of unique exploration strategies that fine-tune the extent of exploration and exploitation based on the statewise variation in radiation exposure during training. Further, we benchmark the predicted path with grid-based deterministic method. We demonstrate that the formulated reward function in conjugation with adequate exploration strategy is effective in handling several scenarios with drastically different radiation field distributions. When compared to vanilla DQN, our model achieves a superior convergence rate and higher training stability.

show abstract

Deep Q-Learning for Two-Hop Communications of Drone Base Stations

Cited by 12 publications

References 31 publications

Reinforcement Learning in the Sky: A Survey on Enabling Intelligence in NTN-Based Communications

Reinforcement Learning in the Sky: A Survey on Enabling Intelligence in NTN-Based Communications

Deep Q-Learning-Based Transmission Power Control of a High Altitude Platform Station with Spectrum Sharing

An Efficient Computing of Correlated Equilibrium for Cooperative $Q$-Learning-Based Multi-Robot Planning

Contact Info

Product

Resources

About