Continuous Control of a Soft Continuum Arm using Deep Reinforcement Learning

Satheeshbabu, Sreeshankar; Uppalapati, Naveen Kumar; Fu, Tianshi; Krishnan, Girish

doi:10.1109/robosoft48309.2020.9116003

Cited by 43 publications

(23 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Q -learning controller proposed in this work uses reinforcement learning to control the soft arm and its environment as a whole. You et al (2017), Satheeshbabu et al (2019, 2020), and Wu et al (2020) use reinforcement learning to implement the control of soft arms under influence of unknown environments or hardware failures. By contrast, our proposed method to increase data by generating virtual goals allows the Q -learning controller to be updated quickly, as a result, our Q -learning controller can deal with greater environmental influence and uncertainty.…”

Section: Discussion and Future Workmentioning

confidence: 99%

“…To better deal with this situation, the states need to contain variables representing the absolute position. For example, Satheeshbabu et al (2020) added the current actuation of the arm to the relative states. However, this will greatly increase the size of the state space, which will mean the controller requires more training data.…”

Section: Discussion and Future Workmentioning

confidence: 99%

“…Satheeshbabu et al (2019) used a Deep Q -Network (DQN) method to implement open-loop positional control of the tip of a soft arm in 3D spaces. And in their later work (Satheeshbabu et al, 2020), deep deterministic policy gradients (DDPGs) were used to implement control of the soft arm with continuous states and actions, and task space feedback was used to improve the ability to handle unknown load. Wu et al (2020) used DQN to implement a position controller of a soft arm in a vertical 2D plane.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Hierarchical control of soft manipulators towards unstructured interactions

Jiang

Wang

Jin

et al. 2021

The International Journal of Robotics Research

View full text Add to dashboard Cite

Performing daily interaction tasks such as opening doors and pulling drawers in unstructured environments is a challenging problem for robots. The emergence of soft-bodied robots brings a new perspective to solving this problem. In this paper, inspired by humans performing interaction tasks through simple behaviors, we propose a hierarchical control system for soft arms, in which the low-level controller achieves motion control of the arm tip, the high-level controller controls the behaviors of the arm based on the low-level controller, and the top-level planner chooses what behaviors should be taken according to tasks. To realize the motion control of the soft arm in interacting with environments, we propose two control methods. The first is a feedback control method based on a simplified Jacobian model utilizing the motion laws of the soft arm that are not affected by environments during interaction. The second is a control method based on [Formula: see text]-learning, in which we present a novel method to increase training data by setting virtual goals. We implement the hierarchical control system on a platform with the Honeycomb Pneumatic Networks Arm (HPN Arm) and validate the effectiveness of this system on a series of typical daily interaction tasks, which demonstrates this proposed hierarchical control system could render the soft arms to perform interaction tasks as simply as humans, without force sensors or accurate models of the environments. This work provides a new direction for the application of soft-bodied arms and offers a new perspective for the physical interactions between robots and environments.

show abstract

Section: Discussion and Future Workmentioning

confidence: 99%

Section: Discussion and Future Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Hierarchical control of soft manipulators towards unstructured interactions

Jiang

Wang

Jin

et al. 2021

The International Journal of Robotics Research

View full text Add to dashboard Cite

show abstract

“…In contrast to Q-learning, SARSA employs the same policy to sample and optimize, which is an onpolicy learning algorithm. In the study by Satheeshbabu et al (2020), the BR 2 soft continuum arm was improved with vision feedback and deep deterministic policy gradient (DDPG), which is a family of actor-critic algorithms. Compared to the previous open-loop control scheme, the closed-loop control method could not only decrease the error obviously but also enable the soft manipulator to track the relatively complex curve path.…”

Section: Reinforcement Learning Without Kinematics/ Dynamics Modelmentioning

confidence: 99%

“…In contrast to policy search reinforcement learning, valuebased methods generate the optimal control policy by optimizing the value function, including SARSA (Ansari et al, 2017b), Q-learning (You et al, 2017;Jiang et al, 2021), DQN (Satheeshbabu et al, 2019;Wu et al, 2020) and its various extensions (e.g., DDQN (You et al, 2019) and Double DQN). The actor-critic approach is a combination of policy-based and value-based reinforcement learning, where the actor executes referring to the policy; thereby the critic calculates the value function to evaluate the actor (Satheeshbabu et al, 2020). Some algorithms (Satheeshbabu et al, 2019;Satheeshbabu et al, 2020;Wu et al, 2020) can be regarded as deep reinforcement learning, which means complex deep neural networks were applied in the control policy, rather than a simple state-action-reward table.…”

Section: Policy-based Vs Value-based Reinforcement Learningmentioning

confidence: 99%

A Survey for Machine Learning-Based Control of Continuum Robots

Wang

Kwok

2021

Front. Robot. AI

View full text Add to dashboard Cite

Soft continuum robots have been accepted as a promising category of biomedical robots, accredited to the robots’ inherent compliance that makes them safely interact with their surroundings. In its application of minimally invasive surgery, such a continuum concept shares the same view of robotization for conventional endoscopy/laparoscopy. Different from rigid-link robots with accurate analytical kinematics/dynamics, soft robots encounter modeling uncertainties due to intrinsic and extrinsic factors, which would deteriorate the model-based control performances. However, the trade-off between flexibility and controllability of soft manipulators may not be readily optimized but would be demanded for specific kinds of modeling approaches. To this end, data-driven modeling strategies making use of machine learning algorithms would be an encouraging way out for the control of soft continuum robots. In this article, we attempt to overview the current state of kinematic/dynamic model-free control schemes for continuum manipulators, particularly by learning-based means, and discuss their similarities and differences. Perspectives and trends in the development of new control methods are also investigated through the review of existing limitations and challenges.

show abstract

Toward Bionic Arthroscopy: A Comprehensive Perspective of Continuum Robotics Principles and Prototypes for the Inception of Arthroscopic Surgery Robots

Huang,

Shi,

Gao

et al. 2023

Advanced Intelligent Systems

View full text Add to dashboard Cite

In this study, an innovative exploration of leveraging bionics and continuum robotics principles to develop a novel solution for arthroscopic surgery is embarked on. Inspired by the flexibility and adaptability of organisms like snakes and octopuses, the continuum robot concept aims to address the inherent challenges in traditional arthroscopy, including lower precision, manual tremors, and long surgeon learning curves. The implementation of these principles in the human body, however, faces significant obstacles, particularly achieving high‐performance motion control amid strong nonlinearity and coupling between modules. This research focuses on intelligent integration and enhanced safety in human‐machine interaction, aiming for improved control precision and flexibility in arthroscopic procedures. A thorough literature review of endoscopic continuum robots is conducted, highlighting current advancements in actuation, structure, sensing, and control technologies. The study concludes with an assessment of these technologies, their limitations, and future potential, in light of the unique demands of arthroscopic continuum robots. This comprehensive review bridges bionics and robotics, presenting the opportunities and challenges in applying continuum robotics to arthroscopic surgery. The goal is to encourage further research in this area, contributing to the development of prototype robots that enhance the precision and safety of arthroscopic surgery.

show abstract

Continuous Control of a Soft Continuum Arm using Deep Reinforcement Learning

Cited by 43 publications

References 17 publications

Hierarchical control of soft manipulators towards unstructured interactions

Hierarchical control of soft manipulators towards unstructured interactions

A Survey for Machine Learning-Based Control of Continuum Robots

Toward Bionic Arthroscopy: A Comprehensive Perspective of Continuum Robotics Principles and Prototypes for the Inception of Arthroscopic Surgery Robots

Contact Info

Product

Resources

About