A Parametric Study of a Deep Reinforcement Learning Control System Applied to the Swing-Up Problem of the Cart-Pole

Abstract

In this investigation, the nonlinear swing-up problem associated with the cart-pole system modeled as a multibody dynamical system is solved by developing a deep Reinforcement Learning (RL) controller. Furthermore, the sensitivity analysis of the deep RL controller applied to the cart-pole swing-up problem is carried out. To this end, the influence of modifying the physical properties of the system and the presence of dry friction forces are analyzed employing the cumulative reward during the task. Extreme limits for the modifications of the parameters are determined to prove that the neural network architecture employed in this work features enough learning capability to handle the task under modifications as high as 90% on the pendulum mass, as well as a 100% increment on the cart mass. As expected, the presence of dry friction greatly affects the performance of the controller. However, a post-training of the agent in the modified environment takes only thirty-nine episodes to find the optimal control policy, resulting in a promising path for further developments of robust controllers.

1.2 nJ/classification 2.4 mm2 asynchronous wired-logic DNN processor using synthesized nonlinear function blocks in 0.18 μm CMOS

“…9(a). 31,32) The weight coefficient was fixed at +1 and the bit width of data was fixed at 8. Since this was a fulldigital implementation, the entire neural net was written in HDL, and the layout could be created only by synthesis and placement and wiring operations.…”

Deep Learning Control for Digital Feedback Systems: Improved Performance with Robustness against Parameter Change

“…RL thereby achieves long-term results, which are otherwise very difficult to achieve. Deep RL has recently been used in robotic manipulation controllers [13,14]. A deep learning controller based on RL is also implemented in [15] for the application of DL in industrial process control.…”

Reinforcement learning approach to control an inverted pendulum: A general framework for educational purposes

“…Many numerical studies have implemented an inverted pendulum virtual environment as a benchmark to test RL algorithms [ 15 – 22 ], but to our knowledge, there is no study that provides successful RL implementations in experiments. First, except for a few studies that have discussed non ideal systems [ 16 , 17 ], most of these numerical implementations discard the effects associated to realistic (and thus more complex) control methods: in experiments, the control of the cart is subject to delay, hysteresis, biases and noise that can significantly alter the learning process. Second, most of the existing virtual environments consider only motion of the pendulum in a small angle range around the upward and unstable position and do not treat the whole control from the downward and stable position as expected in experiments.…”