Setting up a Reinforcement Learning Task with a Real-World Robot

Abstract

Through many recent successes in simulation, model-free reinforcement learning has emerged as a promising approach to solving continuous control robotic tasks. The research community is now able to reproduce, analyze and build quickly on these results due to open source implementations of learning algorithms and simulated benchmark tasks. To carry forward these successes to real-world applications, it is crucial to withhold utilizing the unique advantages of simulations that do not transfer to the real world and experiment directly with physical robots. However, reinforcement learning research with physical robots faces substantial resistance due to the lack of benchmark tasks and supporting source code. In this work, we introduce several reinforcement learning tasks with multiple commercially available robots that present varying levels of learning difficulty, setup, and repeatability. On these tasks, we test the learning performance of off-the-shelf implementations of four reinforcement learning algorithms and analyze sensitivity to their hyper-parameters to determine their readiness for applications in various real-world tasks. Our results show that with a careful setup of the task interface and computations, some of these implementations can be readily applicable to physical robots. We find that state-of-the-art learning algorithms are highly sensitive to their hyper-parameters and their relative ordering does not transfer across tasks, indicating the necessity of re-tuning them for each task for best performance. On the other hand, the best hyper-parameter configuration from one task may often result in effective learning on held-out tasks even with different robots, providing a reasonable default. We make the benchmark tasks publicly available to enhance reproducibility in real-world reinforcement learning 1 .

Learning to Walk Via Deep Reinforcement Learning

“…In particular, we extend the framework of maximum entropy RL. Methods of this type, such as soft actorcritic [17] and soft Q-learning [15], can achieve state-of-the-art sample efficiency [17] and have been successfully deployed in real-world manipulation tasks [16,31], where they exhibit a high degree of robustness due to entropy maximization [16]. However, maximum entropy RL algorithms are sensitive to the choice of the temperature parameter, which determines the trade-off between exploration (maximizing the entropy) and exploitation (maximizing the reward).…”

A Systematic Review on Reinforcement Learning-Based Robotics Within the Last Decade

“…Figure 3 shows a visual representation of the RL algorithm classification and the mathematical description of a few RL algorithms shown in Figure 3 has been described in the appendix section. In comparison model-based RL with model-free RL, model-free RL has proved itself as a promising approach in the field of robotics [31]. Table 3 presents a quick view on the various RL algorithms used in the development of robots in different studies.…”

Automatische Programmierung von Produktionsmaschinen

“…Diese Datenmengen lassen sich mit dem Training auf realen Systemen -also ohne die Verwendung von Simulationen -nur schwer erzeugen. Limitierende Faktoren sind hierbei insbesondere die Voraussetzung einer großen Menge an parallelen Aufbauten, der hohe Zeitaufwand und der Verschleiß der Hardware (Irpan 2020;Hessel et al 2017;Kober et al 2013;Mahmood et al 2018). Aus diesem Grund sind hochqualitative Simulationen zum Training der Algorithmen erforderlich.…”

“…Darüber hinaus steigert diese Art der Initialisierung die benötigten Datenmengen, da der Algorithmus alles von Grund auf lernen muss. Leider ist es insbesondere bei den häufig verwendeten modellfreien Algorithmen nicht möglich, effektiv Vorwissen einzubringen oder bereits gelerntes Wissen auf ähnliche Probleme zu übertragen, was diesem Problem entgegenwirken könnte (Polydoros und Nalpantidis 2017;Mahmood et al 2018;Schulman et al 2015).…”