Development of a Simulator for Prototyping Reinforcement Learning-Based Autonomous Cars

Abstract

Autonomous driving is a research field that has received attention in recent years, with increasing applications of reinforcement learning (RL) algorithms. It is impractical to train an autonomous vehicle thoroughly in the physical space, i.e., the so-called ’real world’; therefore, simulators are used in almost all training of autonomous driving algorithms. There are numerous autonomous driving simulators, very few of which are specifically targeted at RL. RL-based cars are challenging due to the variety of reward functions available. There is a lack of simulators addressing many central RL research tasks within autonomous driving, such as scene understanding, localization and mapping, planning and driving policies, and control, which have diverse requirements and goals. It is, therefore, challenging to prototype new RL projects with different simulators, especially when there is a need to examine several reward functions at once. This paper introduces a modified simulator based on the Udacity simulator, made for autonomous cars using RL. It creates reward functions, along with sensors to create a baseline implementation for RL-based vehicles. The modified simulator also resets the vehicle when it gets stuck or is in a non-terminating loop, making it more reliable. Overall, the paper seeks to make the prototyping of new systems simple, with the testing of different RL-based systems.

A Survey of Vehicle Dynamics Modeling Methods for Autonomous Racing: Theoretical Models, Physical/Virtual Platforms, and Perspectives

“…MORSE (Fig. 10d) [139], jointly developed by LAAS and ONERA, is built on Blender software and exploits the Bullet [141] Unreal Unreal TP Yes 2017 AutoDRIVE [142] PhysX Unity -Yes 2020 LGSVL [143] Unity Unity CM Yes 2020 Grand Theft Auto V [144] Self-developed Self-developed TP No 2013 Gran Turismo Sport [145] Self-developed Self-developed TP No 2017 TORCS [146] Self-developed Self-developed SBM and TP Yes 2007 CarSim [147] Self-developed Relying on display system CM, SBM, and TP No 1990 CarMaker [148] Self-developed Relying on display system CM, SBM, and TP No 1998 rFpro [149] Self-developed Self-developed CM, SBM, and TP No 2007 Cognata [150] Self physics engine. Offering extensive configuration options for vehicle dynamics models and parameters, MORSE enhances the flexibility of simulation platforms.…”

“…10. User interface of the virtual platforms: (a) Gazebo [157]; (b) CoppeliaSim [136]; (c) Webots [158]; (d) MORSE [139]; (e) DeepDrive [159]; (f) CARLA [141]; (g) AutoDRIVE [142]; (h) LGSVL [143]; (i) Grand Theft Auto V [144]; (j) Gran Turismo Sport [145]; (k) TORCS [160]; (l) CarSim [161]; (m) CarMaker [148]; (n) rFpro [149]; (o) VRXPERIENCE [162].…”

“…rFpro (Fig. 10n) [149], owned by AB Dynamics PLC, originated as a project of a Formula One team and is utilized for vehicle dynamics simulation. Focusing on mainstream racetracks, rFpro provides high-fidelity digital models and supports the development, training, testing, and validation of autonomous driving systems.…”