Policy Gradient for s-Rectangular Robust Markov Decision Processes

Abstract

We present a novel robust policy gradient method (RPG) for s-rectangular robust Markov Decision Processes (MDPs). We are the first to derive the adversarial kernel in a closed form and demonstrate that it is a one-rank perturbation of the nominal kernel. This allows us to derive an RPG that is similar to the one used in non-robust MDPs, except with a robust Q-value function and an additional correction term. Both robust Q-values and correction terms are efficiently computable, thus the time complexity of our method matches that of non-robust MDPs, which is significantly faster compared to existing black box methods.

Solving Non-rectangular Reward-Robust MDPs via Frequency Regularization

“…In fact, except for those considered in (Mannor, Mebel, and Xu 2016;Goyal and Grand-Clement 2023) which are locally coupled, s-rectangular uncertainty sets represent the largest class of tractable RMDPs. On the other hand, if not the studies (Xu and Mannor 2010;Mannor, Mebel, and Xu 2016;Derman, Geist, and Mannor 2021;Kumar et al 2023) that treat both reward and transition uncertainty, RMDP literature has mostly focused just on transition uncertainty. We believe this is due to the greater challenge it represents, as the repercussions of transition ambiguity are epistemic and can lead to a butterfly effect: a small kernel deviation at some state can have an unpredictable effect on another state so we are no longer able to track how local kernel uncertainty propagates across the state space.…”

“…In that respect, the robust policy gradient methods recently introduced in (Wang and Zou 2022; Kumar et al 2023;Li, Zhao, and Lan 2022) assume the uncertainty set to be rectangular. Although Wang and Zou (2022) did prove convergence in the non-rectangular case, their analysis exclusively focused on transition uncertainty while they assumed oracle access to the policy gradient.…”

“…In this regard, we introduce the following result: Proposition 12. (Lemma 1 of (Kumar et al 2023)) For all policies π and kernels P , the iterative sequence given by…”