Benchmarking model-free and model-based optimal control

Abstract

Model-free reinforcement learning and nonlinear model predictive control are two different approaches for controlling a dynamic system in an optimal way according to a prescribed cost function. Reinforcement learning acquires a control policy through exploratory interaction with the system, while nonlinear model predictive control exploits an explicitly given mathematical model of the system. In this article, we provide a comprehensive comparison of the performance of reinforcement learning and nonlinear model predictive control for an ideal system as well as for a system with parametric and structural uncertainties. The comparison is based on two different criteria, namely the similarity of trajectories and the resulting rewards. The evaluation of both methods is performed on a standard benchmark problem: a cart-pendulum swing-up and balance task. We first find suitable mathematical formulations and discuss the effect of the differences in the problem formulations. Then, we investigate the robustness of reinforcement learning and nonlinear model predictive control against uncertainties. The results demonstrate that nonlinear model predictive control has advantages over reinforcement learning if uncertainties can be eliminated through identification of the system parameters. Otherwise, there exists a break-even point after which model-free reinforcement learning performs better than nonlinear model predictive control with an inaccurate model. These findings suggest that benefits can be obtained by combining these methods for real systems being subject to such uncertainties. In the future, we plan to develop a hybrid controller and evaluate its performance on a real seven-degree-of-freedom walking robot.

Model-Plant Mismatch Compensation Using Reinforcement Learning

“…The longer prediction horizon used by CAL γ = 0.99 attains a better performance. Another problem of CAL is the slow convergence which is probably caused by the fact that the reward constructed from the quadratic objective function of the nominal controller results in small gradients [2]. This hypothesis is supported by the fact that DPG with a quadratic cost function learns the task extremely slowly.…”

“…Usually, the dynamics of physical systems are known, but various uncertainties do not allow achieving optimal performance with model-based control methods [2]. Whereas for the estimation of parametric uncertainties moving horizon estimation techniques [3] can often be employed, for structural uncertainties, such as backlash, Coulomb friction or wear and tear, this is not easily possible.…”

Control-Based 4D Printing: Adaptive 4D-Printed Systems

“…Then, a policy-based reinforcement-learning feedback controller can be used to learn the nonlinear model of a 4D-printed soft robot through experiments and simulation data to compensate for the uncertainties. The self-learning algorithms play a significant role in adaptive 4D-printed systems to optimise the controller commands based on the information acquired from the interaction with the environment via the 3D-printed sensors [138][139][140][141][142].…”

Trajectory Tracking Control of Robot Manipulators Based on U-Model

“…Nowadays, most robot manipulators need to possess the capacity of accurate and fast trajectory tracking. Trajectory tracking control is a key issue in the field of robot manipulator motion planning [1][2][3]. It aims to enable the joints or links of the robot manipulator to track the desired trajectory with ideal dynamic quality or to stabilize them in the specified position [4].…”