Bridging the Reality Gap Between Virtual and Physical Environments Through Reinforcement Learning

Abstract

Creating Reinforcement learning(RL) agents that can perform tasks in the real-world robotic systems remains a challenging task due to inconsistencies between the virtual-and the real-world. This is known as the ''reality-gap'' which hinders the performance of a RL agent trained in a virtual environment. The research describes the techniques used to train the models, generate randomized environments, reward function, and techniques utilized to transfer the model to the physical environment for evaluation. For this investigation, a low-cost 3-degrees-of-freedom (DOF) Steward platform was 3D modeled and created virtually and physically. The goal of the 3D-Stewart platform was to guide and balance the marble towards the center. Custom end-to-end APIs were developed to interact with the Godot game engine, manipulate physics and dynamics, interact with the in-game lighting and perform environment randomizations. Two RL algorithms: Q-learning and Actor-Critic, were implemented to evaluate the performance by using domain randomization and induced noise to bridge the reality gap. For Q-learning, raw frames were used to make predictions while Actor-Critic utilized marble position, velocity vector and relative position by preprocessing captured frames. The experimental results show the effectiveness of domain randomization and introduction of noise during the training.

Deep Reinforcement Learning with Godot Game Engine

“…The Godot game engine provides a versatile and accessible platform that can be used to train DRL models. This framework was utilized in a recent study [29] to bridge the reality gap between virtual reality and reality using a 3-DoF Stewart platform. This research demonstrates the versatility and efficacy of using Godot game engines in DRL applications.…”

“…In this case, the environment is created in a game engine, which allows the randomization of attributes such as simulated fidelity, physics dynamics, lighting conditions, specular highlights, textures, and object positions and their orientation [22]. These augmentations in the virtual environment create complexity and diversity, allowing the agent to adjust its behavior progressively over time [29].…”

“…The Python framework and Godot game engine were used to perform Sim2Real on a Stewart platform to overcome this gap in reality. In that paper [29], a Godot environment was used to train the DRL algorithm and perform DR to increase the fidelity of the model. The trained DRL model was used to manipulate a real-world 3-DoF Stewart platform.…”

“…The performance results obtained using the framework, demonstrating the application of DRL using Actor-Critic (A) and Q-Learning (B) within different environment conditions. Adapted from[29].…”

Cascaded Fuzzy Reward Mechanisms in Deep Reinforcement Learning for Comprehensive Path Planning in Textile Robotic Systems

“…Additionally, Iriondo A et al employed the Twin Delayed Deep Deterministic Policy Gradient (TD3) method [17] to study the operation of picking up objects from a table using a mobile manipulator. Ranaweera M and colleagues enhanced training outcomes through domain randomization and the introduction of noise during the reinforcement learning process [18]. These methods share a core principle of incorporating probabilistic approaches to significantly reduce the impact of ineffective actions.…”