Open Loop Position Control of Soft Continuum Arm Using Deep Reinforcement Learning

Abstract

Soft robots undergo large nonlinear spatial deformations due to both inherent actuation and external loading. The physics underlying these deformations is complex, and often requires intricate analytical and numerical models. The complexity of these models may render traditional modelbased control difficult and unsuitable. Model-free methods offer an alternative for analyzing the behavior of such complex systems without the need for elaborate modeling techniques. In this paper, we present a model-free approach for open loop position control of a soft spatial continuum arm, based on deep reinforcement learning. The continuum arm is pneumatically actuated and attains a spatial workspace by a combination of unidirectional bending and bidirectional torsional deformation. We use Deep-Q Learning with experience replay to train the system in simulation. The efficacy and robustness of the control policy obtained from the system is validated both in simulation and on the continuum arm prototype for varying external loading conditions.

Comparison and experimental validation of predictive models for soft, fiber-reinforced actuators

“…A study such as this one presented here would, of course, be impossible to perform on all permutations of articulated FREE-based systems. This is where our work complements existing papers such as those of Sadati et al (2017), Satheeshbabu (2018), Thuruthel et al (2019), McMahan et al (2006), and others. The broad, actuator-level understanding presented here can help roboticists in various phases of soft system development and validation: our demonstration of the strengths, weaknesses, and failure points of each of these models can guide model choice for articulated systems of FREEs or soft actuators similar to FREEs, guide design as roboticists find which schemes best suit their modeling capabilities, and inform the development of a next generation of improved soft actuator models.…”

“…Sünderhauf et al (2018) note that deep learning techniques have "been shown to learn predictive physics-based models. " Bruder et al (2018a) show the potential of the Koopman operator in modeling parallel structures of FREEs and Satheeshbabu (2018) uses deep reinforcement learning to model a manipulator made from FREEs.…”

“…Other data-driven methods relying on fundamentally different mathematical structures may be better at capturing the FREE’s underlying physics. Sünderhauf et al (2018) note that deep learning techniques have “been shown to learn predictive physics-based models.”Bruder et al (2018a) show the potential of the Koopman operator in modeling parallel structures of FREEs and Satheeshbabu (2018) uses deep reinforcement learning to model a manipulator made from FREEs. Further, ensemble-based methods, such as those proposed in Perrone and Cooper (1992), might be helpful in addressing network overfitting.…”

Hierarchical control of soft manipulators towards unstructured interactions

“…In our previous work (You et al, 2017), control of the tip of the soft arm is implemented in a 2D plane based on Q -learning and its robustness to destruction of actuators is demonstrated. Satheeshbabu et al (2019) used a Deep Q -Network (DQN) method to implement open-loop positional control of the tip of a soft arm in 3D spaces. And in their later work (Satheeshbabu et al, 2020), deep deterministic policy gradients (DDPGs) were used to implement control of the soft arm with continuous states and actions, and task space feedback was used to improve the ability to handle unknown load.…”

“…For example, You et al (2017) collected data on a physical platform, which costs a lot. In order to expedite training, Satheeshbabu et al (2019) used a mathematical model presented in Uppalapati and Krishnan (2021) to generate virtual training data. Wu et al (2020) used a similar method to generate training data.…”