Robust Value Iteration for Continuous Control Tasks

Abstract

When transferring a control policy from simulation to a physical system, the policy needs to be robust to variations in the dynamics to perform well. Commonly, the optimal policy overfits to the approximate model and the corresponding statedistribution, often resulting in failure to trasnfer underlying distributional shifts. In this paper, we present Robust Fitted Value Iteration, which uses dynamic programming to compute the optimal value function on the compact state domain and incorporates adversarial perturbations of the system dynamics. The adversarial perturbations encourage a optimal policy that is robust to changes in the dynamics. Utilizing the continuoustime perspective of reinforcement learning, we derive the optimal perturbations for the states, actions, observations and model parameters in closed-form. Notably, the resulting algorithm does not require discretization of states or actions. Therefore, the optimal adversarial perturbations can be efficiently incorporated in the min-max value function update. We apply the resulting algorithm to the physical Furuta pendulum and cartpole. By changing the masses of the systems we evaluate the quantitative and qualitative performance across different model parameters. We show that robust value iteration is more robust compared to deep reinforcement learning algorithm and the non-robust version of the algorithm. Videos of the experiments are shown at https://sites.google.com/view/rfvi

Robot Learning From Randomized Simulations: A Review

“…Their sim-to-sim evaluation on four MuJoCo tasks showed that agents trained with the suggested adversarial randomization generalize slightly better to domain parameter configurations than agents trained with a static randomization scheme. Lutter et al (2021a) derived the optimal policy together with different optimal disturbances from the value function in a continuous state, action, and time RL setting. Despite outstanding sim-to-real transferability of the resulting policies, the presented approach is conceptually restricted by assuming access to a compact representation of the state domain, typically obtained through exhaustive sampling, which hinders the scalability to high-dimensional tasks.…”

Optimum UAV Trajectory Design for Data Harvesting From Distributed Nodes

“…This is important because the location component of system states in our DP framework are continuous variables. For continuous-state DP problems, two common approaches for solving the problem numerically are either to approximate the optimum cost functions (e.g., by using least-squares regression or neural-networks) or to discretize the state space [40]- [42].…”

Kernel-based diffusion approximated Markov decision processes for autonomous navigation and control on unstructured terrains

“…The third technique, neural-fitted value function for policy iteration (N-FVPI) , represents a class of value-based RL methods, where a neural network is used to represent the value function v π ( s ) to handle the continuous state space (Heess et al, 2015). During the policy evaluation step, the value function’s parameters are optimized to reduce the one-step squared Bellman residual via gradient descent (Lutter et al, 2021). Like the previous approaches, the policy is implicitly derived by selecting an action that maximizes the Bellman equation in equation (3).…”