Sim–Real Mapping of an Image-Based Robot Arm Controller Using Deep Reinforcement Learning

Abstract

Models trained with Deep Reinforcement learning (DRL) have been deployed in various areas of robotics with varying degree of success. To overcome the limitations of data gathering in the real world, DRL training utilizes simulated environments and transfers the learned policy to real-world scenarios, i.e., sim–real transfer. Simulators fail to accurately capture the entire dynamics of the real world, so simulation-trained policies often fail when applied to reality, termed a reality gap (RG). In this paper, we propose a search (mapping) algorithm that takes in real-world observation (images) and maps them to the policy-equivalent images in the simulated environment using a convolution neural network (CNN) model. The two-step training process, DRL policy and a mapping model, overcomes the RG problem with simulated data only. We evaluated the proposed system with a gripping task of a custom-made robot arm in the real world and compared the performance against a conventional DRL sim–real transfer system. The conventional system achieved a 15–57% success rate in gripping operation depending on the position of the target object while the mapping-based sim–real system achieved 100%. The experimental results demonstrated that the proposed DRL with mapping method appropriately corresponded the real world to the simulated environment, confirming that the scheme can achieve high sim–real generalization at significantly low training costs.

Aplicações de Aprendizado por Reforço em Manipuladores Robóticos: Uma Revisão Sistemática

“…[48,17,19,38] Não utilizou [3,27,24,35,45,20,42,18,47,14,49] Verifica-se também que a variedade de manipuladores robóticos utilizados é ampla, sendo o modelo UR3 da Universal Robots o mais utilizado entre estes em trabalhos com enfoque em: Tarefa peg-in-hole [6,4] e controle de brac ¸o duplo robótico [23]; seguido do PANDA [10,34], UR5 [31,22], RM-X52 [32,33] e IRB 1600 [1,2]. Além disso, 3 trabalhos fizeram o uso de manipuladores produzidos em laboratório, customizados ou com pec ¸as impressas em 3D, implementados em: Controle de articulac ¸ões robóticas [36], planejamento de movimento [46] e mapeamento de controlador de brac ¸o robótico [37]. Já 10,53% dos artigos não informaram o modelo de manipulador implementado no trabalho.…”

“…Outra importante aplicac ¸ão de Q-Learning, em conjunto com DRL foi executada por [37] no mapeamento Sim-Real de um controlador de brac ¸o robótico baseado em imagem, onde foi feita uma comparac ¸ão entre um sistema de transferência de aprendizado convencional DRL e o método com mapeamento proposto. Concluiu-se que o sistema proposto obteve uma taxa de sucesso de 100% na tarefa de preensão, sendo superior ao sistema convencional que obteve entre 15 e 57% de taxa de sucesso a depender da posic ¸ão do objeto.…”

Vibration and Position Control of a Two-Link Flexible Manipulator Using Reinforcement Learning

“…However, when implementing these learned policies on the actual machine, there is a possibility of encountering issues related to the motor's ability to follow the desired trajectories. Previous research has indicated that subtle differences between the numerical simulation environment and the real-world experimental environment can significantly impact the learning outcomes in a negative manner, what is referred to as sim-real challenges [18,38,39].…”

“…In a previous study, we proposed a search algorithm that employed convolutional neural networks to map real-world observations (images) to policy-equivalent images trained by RL in a simulated environment [18]. The system was trained in two steps, involving RL policy and a mapping model, which mitigated the challenges associated with sim-real transfer using solely simulated data.…”

Deep reinforcement learning based voltage control revisited

“…The RG refers to the discrepancy between the simulated environment and the real-world environment that the agent will ultimately be deployed in. This discrepancy can lead to inaccuracy in the model used in the simulation, which can affect the optimization of the policy learned by the agent [26].…”