Finite-Sample Analysis of Off-Policy Natural Actor-Critic with Linear Function Approximation

Abstract

In this paper, we develop a novel variant of off-policy natural actor-critic algorithm with linear function approximation and we establish a sample complexity of O(ǫ −3 ), outperforming all the previously known convergence bounds of such algorithms. In order to overcome the divergence due to deadly triad in off-policy policy evaluation under function approximation, we develop a critic that employs n-step TD-learning algorithm with a properly chosen n. We present finite-sample convergence bounds on this critic under both constant and diminishing step sizes, which are of independent interest. Furthermore, we develop a variant of natural policy gradient under function approximation, with an improved convergence rate of O(1/T ) after T iterations. Combining the finite sample error bounds of actor and the critic, we obtain the O(ǫ −3 ) sample complexity. We derive our sample complexity bounds solely based on the assumption that the behavior policy sufficiently explores all the states and actions, which is a much lighter assumption compared to the related literature. 2 sample complexity, which is the best known convergence bound in the literature for AC algorithms with function approximation.Novelty in the Critic. Off-policy TD with function approximation is famously [65] known to diverge due to deadly triad. To overcome this difficulty, we employ n-step TD-learning, and show that a proper choice of n naturally achieves convergence, and we present finite-sample bounds under both constant and diminishing stepsizes. To the best of our knowledge, we are the first to design a single time-scale off-policy TD with function approximation with provable finite-sample bounds.Novelty in the Actor. NAC under function approximation was developed in [1] by projecting the Qvalues (gradients) to the lower dimensional space, and this involves the use of the discounted state visitation distribution, which is hard to estimate. We develop a new NAC algorithm for the function approximation setting that is instead based on the solution of a projected Bellman equation [73], which our critic is designed to solve.Exploration through Off-Policy Sampling. We establish the convergence bounds under the minimum set of assumptions, viz.,ergodicity under the behavior policy, which ensures sufficient exploration, and thus resolving challenges faced in on-policy sampling. As a result, learning can be done using a single trajectory of samples generated by the behavior policy, and we do not require constant reset of the system that was introduced in on-policy AC algorithms [1, 75] to ensure exploration. A similar observation about employing off-policy sampling to ensure exploration has been made in the tabular setting in [34].1.2. Related Literature. The two main approaches for learning an optimal policy in an RL problem are value space methods, such as Q-learning, and policy space methods, such as AC. The Q-learning algorithm proposed in [77] is perhaps the most well-known value space method. The asymptotic convergence of Qlearning was established in [1...

Global Optimality and Finite Sample Analysis of Softmax Off-Policy Actor Critic under State Distribution Mismatch

“…In this work, we focus on the optimality of naive actor critic algorithms that do not use second order information. With the help of the Fisher information, the optimality of natural actor critic (Kakade, 2001;Peters and Schaal, 2008;Bhatnagar et al, 2009) is also established in both on-policy settings (Agarwal et al, 2020;Wang et al, 2019;Liu et al, 2020;Khodadadian et al, 2021b) and off-policy settings (Khodadadian et al, 2021a;Chen et al, 2021a). Moreover, Xu et al (2021) establish the convergence to stationary points of an off-policy actor critic with density ratio correction and a fixed sampling distribution.…”

“…By contrast, we work on a general off-policy setting in that at any time step the behavior policy can always be arbitrarily different from the target policy. A weaker trackability of the critic can be obtained with the results from Chen et al (2021b) directly without using our extension (i.e., Theorem 2), as done by Chen et al (2021a); Khodadadian et al (2021a) in their analysis of a natural actor critic. However, since Chen et al (2021b) require both the dynamics of the Markov chain and the update operator to be fixed, Chen et al (2021a); Khodadadian et al (2021a) have to keep both the behavior policy and the target policy (actor) fixed when updating the critic.…”

“…A weaker trackability of the critic can be obtained with the results from Chen et al (2021b) directly without using our extension (i.e., Theorem 2), as done by Chen et al (2021a); Khodadadian et al (2021a) in their analysis of a natural actor critic. However, since Chen et al (2021b) require both the dynamics of the Markov chain and the update operator to be fixed, Chen et al (2021a); Khodadadian et al (2021a) have to keep both the behavior policy and the target policy (actor) fixed when updating the critic. That being said, Chen et al (2021a); Khodadadian et al (2021a) have an inner loop for updating the critic and an outer loop for updating the actor.…”

“…However, since Chen et al (2021b) require both the dynamics of the Markov chain and the update operator to be fixed, Chen et al (2021a); Khodadadian et al (2021a) have to keep both the behavior policy and the target policy (actor) fixed when updating the critic. That being said, Chen et al (2021a); Khodadadian et al (2021a) have an inner loop for updating the critic and an outer loop for updating the actor. For the critic to be sufficiently accurate, the inner loop has to take sufficiently many steps.…”

“…For the critic to be sufficiently accurate, the inner loop has to take sufficiently many steps. Chen et al (2021a); Khodadadian et al (2021a), therefore, have a flavor of bi-level optimization. Further, as long as the steps of the inner loop is finite, the bias from using a learned critic instead of the true value function will not diminish in the limit.…”