Reinforcement and Curriculum Learning for Off-Road Navigation of an UGV with a 3D LiDAR

Abstract

This paper presents the use of deep Reinforcement Learning (RL) for autonomous navigation of an Unmanned Ground Vehicle (UGV) with an onboard three-dimensional (3D) Light Detection and Ranging (LiDAR) sensor in off-road environments. For training, both the robotic simulator Gazebo and the Curriculum Learning paradigm are applied. Furthermore, an Actor–Critic Neural Network (NN) scheme is chosen with a suitable state and a custom reward function. To employ the 3D LiDAR data as part of the input state of the NNs, a virtual two-dimensional (2D) traversability scanner is developed. The resulting Actor NN has been successfully tested in both real and simulated experiments and favorably compared with a previous reactive navigation approach on the same UGV.

Waypoint Generation in Satellite Images Based on a CNN for Outdoor UGV Navigation

“…The computer of Andabata employs an inertial measurement unit (IMU), with inclinometers, gyroscopes, and a compass, and a global navigation satellite system (GNSS) receiver with a horizontal resolution of 1 m included in its onboard smartphone for outdoor localization [36]. The main exteroceptive sensor for navigation is a custom 3D LiDAR sensor with 360°field of view built by rotating a 2D LiDAR [37].…”

“…Although waypoints for the UGV are calculated in the detected paths, reactivity is still necessary to avoid steep slopes and unexpected obstacles that are not visible on satellite images. Local navigation between distant waypoints has been implemented on Andabata with a previously developed actor-critic scheme, which was trained using reinforcement and curriculum learning [36].…”

“…Basically, acquired 3D point clouds are employed to emulate a 2D traversability scanner, which produces 32 virtual levelled ranges up to 10 m around the vehicle (see Figure 19). These data, together with the heading error of the vehicle with respect to the current waypoint (p t ), are employed by the actor neural network to directly produce steering speed commands while moving at a constant longitudinal speed [36]. When the distance to the current waypoint (d t ) is less than 1 m, the next objective from the list is chosen.…”

“…Figure19. Representation of a virtual 2D traversability scan for Andabata[36]. A nearby obstacle is shown in grey.…”

A Concept of a Plug-In Simulator for Increasing the Effectiveness of Rescue Operators When Using Hydrostatically Driven Manipulators

“…Less common, but more relevant from that perspective, especially as computation capabilities increase, is the possibility of having deformable environmental objects and terrain [ 14 ]. Conducted analyses have shown that there are several simulators which offer extensive use case possibilities [ 15 , 16 , 17 , 18 ]. A solution commonly used in research is the MuJoCo [ 19 ].…”