Policy gradient primal-dual mirror descent for constrained MDPs with large state spaces

Abstract

We study the problem of computing an optimal policy of an infinite-horizon discounted constrained Markov decision process (constrained MDP). Despite the popularity of Lagrangian-based policy search methods used in practice, the oscillation of policy iterates in these methods has not been fully understood, bringing out issues such as violation of constraints and sensitivity to hyper-parameters. To fill this gap, we employ the Lagrangian method to cast a constrained MDP into a constrained saddle-point problem in which max/min players correspond to primal/dual variables, respectively, and develop two single-time-scale policy-based primal-dual algorithms with non-asymptotic convergence of their policy iterates to an optimal constrained policy. Specifically, we first propose a regularized policy gradient primal-dual (RPG-PD) method that updates the policy using an entropy-regularized policy gradient, and the dual via a quadratic-regularized gradient ascent, simultaneously. We prove that the policy primal-dual iterates of RPG-PD converge to a regularized saddle point with a sublinear rate, while the policy iterates converge sublinearly to an optimal constrained policy. We further instantiate RPG-PD in large state or action spaces by including function approximation in policy parametrization, and establish similar sublinear last-iterate policy convergence. Second, we propose an optimistic policy gradient primal-dual (OPG-PD) method that employs the optimistic gradient method to update primal/dual variables, simultaneously. We prove that the policy primal-dual iterates of OPG-PD converge to a saddle point that contains an optimal constrained policy, with a linear rate. To the best of our knowledge, this work appears to be the first non-asymptotic policy last-iterate convergence result for single-time-scale algorithms in constrained MDPs. We further exhibit the merits and effectiveness of our methods in computational experiments.

Recent advances in reinforcement learning in finance

“…(2020) developed a martingale approach to learn policies that are sensitive to the uncertainty of the rewards and are meaningful under some market scenarios. Another line of work focuses on constrained RL problems with different risk criteria (Achiam et al., 2017; Chow et al., 2017, 2015; Ding et al., 2021; Tamar et al., 2015; Zheng & Ratliff, 2020). Very recently, Jaimungal et al.…”

TRC: Trust Region Conditional Value at Risk for Safe Reinforcement Learning

“…Then, a suboptimal policy is obtained by iteratively solving the subproblem (8) with linear approximations on the objective and the safety constraint and quadratic approximations on the KL divergence term.…”

“…Thus, we approximate the upper bound on CVaR by removing the divergence term in ( 14) and add a trust-region constraint as in [12] and [15]. Then, the proposed CVaR-constrained subproblem can be written as below by replacing the constraint in (8) with the approximated CVaR.…”