Actor-critic reinforcement learning for tracking control in robotics

Pane, Yudha; Nageshrao, Subramanya; Babuška, Robert

doi:10.1109/cdc.2016.7799164

Cited by 22 publications

(11 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• We extend our initial results from Pane et al (2016). Based on the RL-based control input compensator from Pane et al (2016), in this work a novel RLbased method, called the reference compensation method, is developed.…”

Section: Introductionmentioning

confidence: 87%

See 1 more Smart Citation

Reinforcement learning based compensation methods for robot manipulators

Pane

Nageshrao

Kober

et al. 2019

Engineering Applications of Artificial Intelligence

Self Cite

View full text Add to dashboard Cite

Smart robotics will be a core feature while migrating from Industry 3.0 (i.e., mass manufacturing) to Industry 4.0 (i.e., customized or social manufacturing). A key characteristic of a smart system is its ability to learn. For smart manufacturing, this means incorporating learning capabilities into the current fixed, repetitive, task-oriented industrial manipulators, thus rendering them 'smart'. In this paper we introduce two reinforcement learning (RL) based compensation methods. The learned correction signal, which compensates for unmodeled aberrations, is added to the existing nominal input with an objective to enhance the control performance. The proposed learning algorithms are evaluated on a 6-DoF industrial robotic manipulator arm to follow different kinds of reference paths, such as square or a circular path, or to track a trajectory on a three dimensional surface. In an extensive experimental study we compare the performance of our learning-based methods with well-known tracking controllers, namely, proportional-derivative (PD), model predictive control (MPC), and iterative learning control (ILC). The experimental results show a considerable performance improvement thanks to our RL-based methods when compared to PD, MPC, and ILC.

show abstract

Section: Introductionmentioning

confidence: 87%

“…PD is used as a baseline for model-free, non adaptive control method, while MPC is used as a reference for model-based control framework and finally ILC is chosen as a baseline for model-free and adaptive method. In Pane et al (2016) we only compared against PD.…”

Section: Introductionmentioning

confidence: 99%

Reinforcement learning based compensation methods for robot manipulators

Pane

Nageshrao

Kober

et al. 2019

Engineering Applications of Artificial Intelligence

Self Cite

View full text Add to dashboard Cite

show abstract

“…AC compensators were designed, 120 which was used to reduce tracking error of a multiple DOF industrial robot manipulator. A variety of real-world robotic manipulation tasks, such as dish placement and pouring, used policy optimization to adaptively sample trajectories and effectively to learn good global costs for complex robotic motion skills from user demonstrations.…”

Section: Trajectory and Route Trackingmentioning

confidence: 99%

Reinforcement learning for robot research: A comprehensive review and open issues

Zhang

2021

International Journal of Advanced Robotic Systems

View full text Add to dashboard Cite

Applying the learning mechanism of natural living beings to endow intelligent robots with humanoid perception and decision-making wisdom becomes an important force to promote the revolution of science and technology in robot domains. Advances in reinforcement learning (RL) over the past decades have led robotics to be highly automated and intelligent, which ensures safety operation instead of manual work and implementation of more intelligence for many challenging tasks. As an important branch of machine learning, RL can realize sequential decision-making under uncertainties through end-to-end learning and has made a series of significant breakthroughs in robot applications. In this review article, we cover RL algorithms from theoretical background to advanced learning policies in different domains, which accelerate to solving practical problems in robotics. The challenges, open issues, and our thoughts on future research directions of RL are also presented to discover new research areas with the objective to motivate new interest.

show abstract

“…In ILC, the tracking performance is improved by adjusting control inputs or reference signals in repeated trials (Bristow et al, 2006;Schoellig et al, 2012;Tayebi, 2004). In addition to ILC, reinforcement learning (RL)-based approaches have also been proposed to iteratively optimize the tracking performance (Kiumarsi et al, 2014;Pane et al, 2016;Zhang et al, 2016a). Apart from iterative approaches, there are also various works on improving the tracking performance of classical model-based controllers by learning the uncertain or unknown system dynamics with techniques such as Gaussian processes (GPs) (Helwa et al, 2018;Nguyen-Tuong and Peters, 2008), neural networks (NNs) (He et al, 2016;Yan and Wang, 2014), and support vector machines (SVMs) (Iplikci, 2006).…”

Section: Introductionmentioning

confidence: 99%

“…Unlike the iterative learning methods (ILC approaches and some RL-based approaches such as Pane et al (2016)), the proposed DNN approach can be directly used for tracking arbitrary, feasible trajectories without further adaptations during the testing phase, and, consequently, it satisfies the impromptu tracking requirement. Compared to more common approaches (such as forward or inverse dynamic learning) where the learning component typically resides in the main control loop, we use the DNN module as an add-on block that is placed outside of the closed-loop system to improve the tracking performance.…”

Section: Introductionmentioning

confidence: 99%

Deep neural networks as add-on modules for enhancing robot performance in impromptu trajectory tracking

Zhou

Helwa

Schoellig

2020

The International Journal of Robotics Research

View full text Add to dashboard Cite

High-accuracy trajectory tracking is critical to many robotic applications, including search and rescue, advanced manufacturing, and industrial inspection, to name a few. Yet the unmodeled dynamics and parametric uncertainties of operating in such complex environments make it difficult to design controllers that are capable of accurately tracking arbitrary, feasible trajectories from the first attempt (i.e., impromptu trajectory tracking). This article proposes a platform-independent, learning-based “add-on” module to enhance the tracking performance of black-box control systems in impromptu tracking tasks. Our approach is to pre-cascade a deep neural network (DNN) to a stabilized baseline control system, in order to establish an identity mapping from the desired output to the actual output. Previous research involving quadrotors showed that, for 30 arbitrary hand-drawn trajectories, the DNN-enhancement control architecture reduces tracking errors by 43% on average, as compared with the baseline controller. In this article, we provide a platform-independent formulation and practical design guidelines for the DNN-enhancement approach. In particular, we: (1) characterize the underlying function of the DNN module; (2) identify necessary conditions for the approach to be effective; (3) provide theoretical insights into the stability of the overall DNN-enhancement control architecture; (4) derive a condition that supports data-efficient training of the DNN module; and (5) compare the novel theory-driven DNN design with the prior trial-and-error design using detailed quadrotor experiments. We show that, as compared with the prior trial-and-error design, the novel theory-driven design allows us to reduce the input dimension of the DNN by two thirds while achieving similar tracking performance.

show abstract

Actor-critic reinforcement learning for tracking control in robotics

Cited by 22 publications

References 6 publications

Reinforcement learning based compensation methods for robot manipulators

Reinforcement learning based compensation methods for robot manipulators

Reinforcement learning for robot research: A comprehensive review and open issues

Deep neural networks as add-on modules for enhancing robot performance in impromptu trajectory tracking

Contact Info

Product

Resources

About