Hierarchical Reinforcement Learning Framework Towards Multi-Agent Navigation

Abstract

In this paper, we propose a navigation algorithm oriented to multi-agent environment. This algorithm is expressed as a hierarchical framework that contains a Hidden Markov Model (HMM) and a Deep Reinforcement Learning (DRL) structure. For simplification, we term our method Hierarchical Navigation Reinforcement Network (HNRN). In highlevel architecture, we train an HMM to evaluate the agents perception to obtain a score. According to this score, adaptive control action will be chosen. While in low-level architecture, two sub-systems are introduced, one is a differential targetdriven system, which aims at heading to the target; the other is a collision avoidance DRL system, which is used for avoiding dynamic obstacles. The advantage of this hierarchical structure is decoupling the target-driven and collision avoidance tasks, leading to a faster and more stable model to be trained. The experiments indicate that our algorithm has higher learning efficiency and rate of success than traditional Velocity Obstacle (VO) algorithms or hybrid DRL method.

Deep reinforcement learning based mobile robot navigation: A review

“…To simplify the decision-making process, raw sensor data can be directly used as input. Ding et al [59] studied the hierarchical RL method using lidar data as input. A high-level evaluation module is responsible for perceiving the overall environmental risks, and a low-level control module is responsible for making action decisions.…”

Low-Cost Multi-Agent Navigation via Reinforcement Learning With Multi-Fidelity Simulator

“…For example, Long et al [4] used the PPO algorithm end-to-end training obstacle avoidance algorithm on the Stage simulator. Ding et al [9] trained the DDPG and HMM joint algorithm on the Gazebo simulator [10] to solve the multi-vehicle collaborative path planning. Tai et al [11] trained ADDPG on V-REP [12] to solve the Mapless Navigation problem and Kahn et al [13] trained a model-based RL algorithm on open raves to solve obstacle avoidance.…”

Motion Planning and Control for Mobile Robot Navigation Using Machine Learning: a Survey

“…Both Tai et al (2018) and Jin et al (2019) have shown that systems similar to the ones above can also be used to enable navigation in the presence of human pedestrians using RL and geometric sensor inputs. Ding et al (2018) showed that a similar capability can also be achieved using RL to train a system to choose between target pursuit and collision avoidance by incorporating a Hidden Markov Model (HMM) (Stratonovich 1965) into a hierarchical model. Considering the specific case in which some humans in the scene can be assumed to be companions, Li et al (2018) showed that end-to-end learned approaches could also enable Socially Concomitant Navigation (SCN), i.e., navigation in which the robot not only needs to avoid collisions as in previous work, but also needs to maintain a sense of affinity with respect to the motion of its companion.…”