Provably Efficient Primal-Dual Reinforcement Learning for CMDPs with Non-stationary Objectives and Constraints

Abstract

We consider primal-dual-based reinforcement learning (RL) in episodic constrained Markov decision processes (CMDPs) with non-stationary objectives and constraints, which plays a central role in ensuring the safety of RL in time-varying environments. In this problem, the reward/utility functions and the state transition functions are both allowed to vary arbitrarily over time as long as their cumulative variations do not exceed certain known variation budgets. Designing safe RL algorithms in time-varying environments is particularly challenging because of the need to integrate the constraint violation reduction, safe exploration, and adaptation to the non-stationarity. To this end, we identify two alternative conditions on the time-varying constraints under which we can guarantee the safety in the long run. We also propose the Periodically Restarted Optimistic Primal-Dual Proximal Policy Optimization (PROPD-PPO) algorithm that can coordinate with both two conditions. Furthermore, a dynamic regret bound and a constraint violation bound are established for the proposed algorithm in both the linear kernel CMDP function approximation setting and the tabular CMDP setting under two alternative conditions. This paper provides the first provably efficient algorithm for non-stationary CMDPs with safe exploration.

Robustness Guarantees for Mode Estimation with an Application to Bandits

“…Here the learner has access to a safe policy that can be deployed while the learner does not have sufficient knowledge of the safety constraint. This RL setting has been studied in tabular (Efroni et al, 2020;Liu et al, 2021a;Bura et al, 2022) and linear (Ding et al, 2021;Ghosh et al, 2022) MDPs. It is notable that Liu et al (2021a) make use of the optimism-pessimism principle that we developed in our earlier work (Pacchiano et al, 2021) and used in the analysis of this paper.…”