Autonomous Exploration of Mobile Robots via Deep Reinforcement Learning Based on Spatiotemporal Information on Graph

Abstract

In this paper, we address the problem of autonomous exploration in unknown environments for ground mobile robots with deep reinforcement learning (DRL). To effectively explore unknown environments, we construct an exploration graph considering historical trajectories, frontier waypoints, landmarks, and obstacles. Meanwhile, to take full advantage of the spatiotemporal feature and historical information in the autonomous exploration task, we propose a novel network called Spatiotemporal Neural Network on Graph (Graph-STNN). Specifically, the proposed Graph-STNN extracts the spatial feature using graph convolutional network (GCN) and the temporal feature using temporal convolutional network (TCN). Then, gated recurrent unit (GRU) is performed to synthesize the spatial feature, the temporal feature, and the historical state information into the current state feature. Combined with DRL, our Graph-STNN helps estimation of the optimal target point through extracted hybrid features. The simulation experiment shows that our approach is more effective than the GCN-based approach and the information entropy-based approach. Moreover, Graph-STNN also performs better generalization ability than GCN-based, information entropy-based, and random methods. Finally, we validate our approach on the simulation platform Stage with the actual robot model.

Autonomous exploration through deep reinforcement learning

“…Although the robot is a brief model based on the particle in this environment, the size of the environment is fixed and the environment is highly structured, this is still a brave attempt in the exploration field and has confirmed the possibility of using reinforcement learning algorithms to solve this problem. Zhang et al (2021) and Chen et al (2020) propose a reinforcement learning method using a graph convolution neural network to solve the decision-making problem in exploration and train the policy in many random maps, taking the historical position, current position, observation landmark and candidate frontiers of the robot as the vertices of the graph and connecting these vertices by some rules to form a graph. However, this method still relies on a series of processing on occupancy grid maps such as frontier detection to a certain extent, and when the environmental characteristics change, the efficiency of this graph-based method is not high.…”

Indoor Visual Exploration with Multi-Rotor Aerial Robotic Vehicles

“…In [ 10 ], a Topology–Grid Hybrid Map (TGHM) scheme is formulated for autonomous exploration. Modern approaches, such as Reinforcement Learning (RL), have also been employed to tackle the exploration problem [ 11 ].…”