Statistical Inference of the Value Function for Reinforcement Learning in Infinite-Horizon Settings

Abstract

Reinforcement learning is a general technique that allows an agent to learn an optimal policy and interact with an environment in sequential decision making problems. The goodness of a policy is measured by its value function starting from some initial state. The focus of this paper is to construct confidence intervals (CIs) for a policy's value in infinite horizon settings where the number of decision points diverges to infinity. We propose to model the action-value state function (Q-function) associated with a policy based on series/sieve method to derive its confidence interval. When the target policy depends on the observed data as well, we propose a SequentiAl Value Evaluation (SAVE) method to recursively update the estimated policy and its value estimator. As long as either the number of trajectories or the number of decision points diverges to infinity, we show that the proposed CI achieves nominal coverage even in cases where the optimal policy is not unique. Simulation studies are conducted to back up our theoretical findings. We apply the proposed method to a dataset from mobile health studies and find that reinforcement learning algorithms could help improve patient's health status.

Reinforcement Learning in Possibly Nonstationary Environments

“…2 is upper bounded by some constant. Using similar arguments in Part 3 of the proof of Lemma 3 in Shi et al (2021), we can show that…”

“…Sixth, (A6) and (A7) are commonly imposed in the statistics literature on RL (see e.g., Luckett et al, 2020). ( A6) is automatically satisfied when the behavior policy that determines Shi et al, 2021). ( A7) is a necessary condition for establishing the limiting distribution of…”

“…We next learn the optimal policy using each data subset, evaluate its Q-value on the other dataset based on linear sieves (see e.g. Shi et al, 2021) and construct a CUSUM-type test statistic based on these estimated Q-values. This yields two test statistics, for each of the data subsets.…”

“…The latter setting is widely studied in the computer science literature to formulate many sequential decision making problems in games, robotics, ridesharing, etc. Recently, a few algorithms have been proposed in the statistics literature for policy optimisation or evaluation in infinite horizon settings (Ertefaie and Strawderman, 2018;Luckett et al, 2020;Liao et al, 2020b;Hu et al, 2021a;Liao et al, 2021;Shi et al, 2021;Ramprasad et al, 2021).…”

Off-Policy Fitted Q-Evaluation with Differentiable Function Approximators: Z-Estimation and Inference Theory

“…, where ξ π is the stationary distribution of underlying transition kernel and α is the decay rate of eigenvalues of kernel functions. Moreover, [Shi et al, 2021, Uehara et al, 2021 proposed their estimators for Q-functions derived the L 2 -norm convergence rate. [Chen and Qi, 2022] extended this result in a more general setting under weaker condition.…”