A Hierarchical Reinforcement Learning Framework for UAV Path Planning in Tactical Environments

Abstract

Tackling tactical UAV path planning under radar threat using reinforcement learning involves particular challenges ranging from modeling related difficulties to sparse feedback problem. Learning goal-directed behavior with sparse feedback from complex environments is a fundamental challenge for reinforcement learning algorithms. In this paper we extend our previous work in this area to provide a solution to the problem setting stated above, using Hierarchical Reinforcement Learning (HRL) in a novel way that involves a meta controller for higher level goal assignment and a controller that determines the lower-level actions of the agent. Our meta controller is based on a regression model trained using a state transition scheme that defines the evolution of goal designation, whereas our lower-level controller is based on a Deep Q Network (DQN) and is trained via reinforcement learning iterations. This two-layer framework ensures that an optimal plan for a complex path, organized as multiple goals, is achieved gradually, through piecewise assignment of sub-goals, and thus as a result of a staged, efficient and rigorous procedure.

Generating Function Reallocation to Handle Contingencies in Human–Robot Teaming Missions: The Cases in Lunar Surface Transportation

“…Reinforcement learning is considered as an efficient approach to solve the optimal allocation problem [25]. By learning to operate on different levels of temporal abstraction, hierarchical reinforcement learning (HRL) can solve the reinforcement learning expansion problem, and considering its advantages in data richness complexity, efficient data utilization and learning performance, this paper uses the HRL algorithm with two layers for solving the optimal human-robot functional assignment problem [26,27].…”