Towards Generalization and Simplicity in Continuous Control

Abstract

This work shows that policies with simple linear and RBF parameterizations can be trained to solve a variety of widely studied continuous control tasks, including the OpenAI gym benchmarks. The performance of these trained policies are competitive with state of the art results, obtained with more elaborate parameterizations such as fully connected neural networks. Furthermore, the standard training and testing scenarios for these tasks are shown to be very limited and prone to overfitting, thus giving rise to only trajectory-centric policies. Training with a diverse initial state distribution induces more global policies with better generalization. This allows for interactive control scenarios where the system recovers from large on-line perturbations; as shown in the supplementary video.

Integration of adaptive control and reinforcement learning for real-time control and learning

“…u exog (t) can be determined using u r (t), x r (t), and (37) as φ(x) is a known function of x. It is clear from ( 39) -( 41) that if there are no parametric uncertainties, and if the initial conditions of ( 35) are identical to those of (36), with K(0) = I, Θ nl (0) = Θ nl,r and Θ l (0) = Θ l,r , then the AC-RL policy coincides with u r (t). For the rest of this paper, unless otherwise mentioned, we choose Q = 2I.The following theorem presents the stability property of the AC-RL as well as Regret (defined in ( 14)):…”

“…Theorem 2. Under Assumptions A1-A2, A4', and A5, the closed-loop adaptive system specified by (35), (36), (38) and (43) has globally bounded solutions with lim t→∞ e(t) = 0 with R = O(1).…”

“…Theorem 3. Under Assumptions A1-A3 and A4', the closed-loop adaptive system specified by the target system (35), the reference system (36), the magnitude constraint in (46) and the MSAC-RL controller given by ( 48)-( 51) will (i) have globally bounded solutions, with lim t→∞ e u (t) = 0 if the target system in (35) is open-loop stable.…”

“…As Theorems 1-3 pertain to the same plant (35), reference system (36), and control input in (8), the following derivation is common to their proofs. From Assumption A3 and Eq.…”

“…In practice, however, offline policies trained in simulation often exhibit degenerate performance when used for real-time control due to modeling errors that can occur online [32,33]. It may be difficult to reliably predict the behavior of a learned policy when it is applied to an environment different from the one seen during training [34][35][36][37]. The first set of approaches we propose in this paper seeks to bridge this "sim-to-real" gap by proposing various combinations of the AC and RL approaches so as to realize their individual advantages and minimize their weaknesses.…”