Framework for Armature-Based 3D Shape Reconstruction of Sensorized Soft Robots in eXtended Reality

Abstract

Soft robots are typically intended to operate in highly unpredictable and unstructured environments. Although their soft bodies help them to passively conform to their environment, the execution of specific tasks within such environments often requires the help of an operator that supervises the interaction between the robot and its environment and adjusts the actuation inputs in order to successfully execute the task. However, direct observation of the soft robot is often impeded by the environment in which it operates. Therefore, the operator has to depend on a real-time simulation of the soft robot based on the signals from proprioceptive sensors. However, the complicated three-dimensional (3D) configurations of the soft robot can be difficult to interpret using traditional visualization techniques. In this work, we present an open-source framework for real-time 3D reconstruction of soft robots in eXtended Reality (Augmented and Virtual Reality), based on signals from their proprioceptive sensors. This framework has a Robot Operating System (ROS) backbone, allowing for easy integration with existing soft robot control algorithms for intuitive and real-time teleoperation. This approach is demonstrated in Augmented Reality using a Microsoft Hololens device and runs at up to 60 FPS. We explore the influence that system parameters such as mesh density and armature complexity have on the reconstruction's key performance metrics (i.e., speed, scalability). The open-source framework is expected to function as a platform for future research and developments on real-time remote control of soft robots operating in environments that impede direct observation of the robot.

Towards next generation digital twin in robotics: Trends, scopes, challenges, and future

“…The most recent research trends of DT in soft robotics aims towards the augmented or extended reality. Apart from establishing effective HRI systems, which will be discussed in the next subsection, virtual and extended reality has exhibited promising results in construction, development and functionality of soft robots in recent years [49] , [50] . In 2019, [51] proposed that soft robots be used as virtual humans in a human-centered production systems, where the infrastructure's efficiency may be further increased by leveraging DT in VR.…”

“…A critical challenge to presenting the digital replica of a soft robot is the reconstruction of its shape in the virtual space [49] . Albeit there are several approaches to reconstructing DT of soft robots, when it comes to accuracy most of them cannot suffice.…”

“…Albeit there are several approaches to reconstructing DT of soft robots, when it comes to accuracy most of them cannot suffice. As investigated by [49] , simple approaches such as single bending angle and curvature parameterizations were inadequate to represent the complex data received by proprioceptive sensor as a full-fledged digital model. Again, ease of understanding being a critical priority in any DT interface restricted representation of DT based on a set of 3D coordinate points as the approach leaded to great difficulty.…”

“…However, construction of RDT model, especially for soft robots and exporting that to the VR space was found to be quite a tedious task [49] . Again, as it was discussed in [53] RDTs based on traditional optical sensors are limited to presence of light, which leaves room for further research opportunities to overcome such limitations.…”