From Simulation to Real World Maneuver Execution using Deep Reinforcement Learning

Abstract

Deep Reinforcement Learning has proved to be able to solve many control tasks in different fields, but the behavior of these systems is not always as expected when deployed in real-world scenarios. This is mainly due to the lack of domain adaptation between simulated and real-world data together with the absence of distinction between train and test datasets. In this work, we investigate these problems in the autonomous driving field, especially for a maneuver planning module for roundabout insertions. In particular, we present a system based on multiple environments in which agents are trained simultaneously, evaluating the behavior of the model in different scenarios. Finally, we analyze techniques aimed at reducing the gap between simulated and real-world data showing that this increased the generalization capabilities of the system both on unseen and real-world scenarios.

Driving Forward Mobile Robotics: A Digital Twin Architecture Case Study for AGVs Data-Driven Autonomy

“…This challenge was also identified by [25] and [26] regarding Digital Twin implementations. Despite the identified challenges, the implementation of a modular Digital Twin architecture in the context of mobile robotics can bring benefits in very different scenarios , such as provide faster problem detection due to the capabilities of the technologies used in Digital Twin's services [27], fail-safe sensing for cases where robot hardware devices fail [28], creating this way systems and components redundancy, multi-agv management [29,30], when there is the need to manage and optimize multiple robots routes [31], digitalfirst approach that allows for testing or calibration [32,33] of different hardware and robot design or even validate technologies before creating the physical asset [34]. These potential benefits are described on Table 1.…”

Tackling Real-World Autonomous Driving using Deep Reinforcement Learning

“…Paolo Capasso, Giulio Bacchiani, Alberto Broggi are with Vislab srl, an Ambaraella Inc. company -Parma, Italy acapasso@ambarella.com, gbacchiani@ambarella.com, broggi@vislab.it In particular, RL algorithms are widely used in the autonomous driving field for the development of decisionmaking and maneuver execution systems like lane change ( [11], [12], [13]), lane keeping ( [14], [15]), overtaking maneuvers [16], intersection and roundabout handling ( [17], [18]) and many others. Starting from the delayed version of Asynchronous Advantage Actor Critic (A3C) algorithm ( [19], [20], [21]), we implemented a Reinforcement Learning planner training agents in a simulator based on High Definition Maps (HD Maps [22]) developed internally by the research team. In particular, we trained the model predicting continuous actions related to the acceleration and steering angle, and testing it on board of a real self-driving car on an entire urban area of the city of Parma (Fig.…”

“…In this paper we used a delayed version of the original A3C [19] called Delayed-A3C (D-A3C). This algorithm was previously developed and used in [20] and [21] where it has been shown that it allows to achieve better results than A3C. In D-A3C configuration, each agent begins the episode with a local copy of the latest version of the global network, while the system collects all the contribution of the actors; the agent updates their local copy of the network at fixed time intervals but all the updates are sent to the global network only at the end of the episode, while in the classical A3C algorithm this exchange is performed at fixed time intervals.…”

Virtual to Real-World Transfer Learning: A Systematic Review

“…Logistics Research [38] Science Robotics [39] International Symposium on Experimental Robotics (ISER) [41] IEEE/CVF International Conferene on Computer Vision (ICCV) [43] IEEE Transactions on Vehicular Technology [44] IEEE Robotics and Automation Letters [45] IEEE Intelligent Vehicles Symposium [46] International Conference on Informatics in Control, Automation and Robotics [47] European Conference on Machine Learning [48] International Joint Conference on Artificial Intelligence [49] International Conference on Unsupervised and Transfer Learning workshop [50] International Conference on Neural Information Processing Systems [51] International Conference on Learning Representations (ICLR) Conference [52,54,55] International Conference on Machine Learning [56,57] Journal of Machine Learning Research [58,72] Springer [59] Nature [60] Stanford University AI Lab [62] ACM Transactions on Intelligent Systems and Technology [66] IEEE Transactions on Pattern Analysis and Machine Intelligence [67] AAAI Publications, 2016 AAAI Spring Symposium Series [70] IEEE International Conference on Data Mining Workshops (ICDMW) [71] Robotics and Autonomous Systems [73] Artificial Intelligence [74] Sensors [78] Synthesis lectures on Artificial Intelligence and Machine Learning [79]…”

“…Additionally, it allows a reduction in the size of the datasets which in turn reduces the computational requirements. Examples are Autonomous MAV flight [41], motion planning [29], domain adaptation for improved robot grasping [30,31,35,45], multirobot transfer learning [24,32,53,65,66], mobile fulfilment systems [38] and autonomous driving [42][43][44][45][46][47]. The above mentioned papers use virtual training environments to generate synthetic training data to train a model in the virtual environment and use transfer learning techniques to transfer the knowledge to real-world platforms.…”