Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes

Abstract

We study offline reinforcement learning (RL) in partially observable Markov decision processes. In particular, we aim to learn an optimal policy from a dataset collected by a behavior policy which possibly depends on the latent state. Such a dataset is confounded in the sense that the latent state simultaneously affects the action and the observation, which is prohibitive for existing offline RL algorithms. To this end, we propose the Proxy variable Pessimistic Policy Optimization (P3O) algorithm, which addresses the confounding bias and the distributional shift between the optimal and behavior policies in the context of general function approximation. At the core of P3O is a coupled sequence of pessimistic confidence regions constructed via proximal causal inference, which is formulated as minimax estimation. Under a partial coverage assumption on the confounded dataset, we prove that P3O achieves a n −1/2 -suboptimality, where n is the number of trajectories in the dataset. To our best knowledge, P3O is the first provably efficient offline RL algorithm for POMDPs with a confounded dataset. Contents

Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processes

“…In the meanwhile, to ensure identifiability, Namkoong et al (2020) consider the case where the unmeasured confounders affect only one of the decisions made. Very recently, there is a stream of research focused on using proximal causal inference (Tchetgen et al, 2020) for off-policy evaluation and learning in the partially observed MDP (Bennett et al, 2021;Shi et al, 2021;Lu et al, 2022).…”