Learning Infinite-horizon Average-reward MDPs with Linear Function Approximation

Abstract

We develop several new algorithms for learning Markov Decision Processes in an infinite-horizon average-reward setting with linear function approximation. Using the optimism principle and assuming that the MDP has a linear structure, we first propose a computationally inefficient algorithm with optimal O( √ T ) regret and another computationally efficient variant with O(T 3 4 ) regret, where T is the number of interactions. Next, taking inspiration from adversarial linear bandits, we develop yet another efficient algorithm with O( √ T ) regret under a different set of assumptions, improving the best existing result by Hao et al. [16] with O(T 2 3 ) regret. Moreover, we draw a connection between this algorithm and the Natural Policy Gradient algorithm proposed by Kakade [22], and show that our analysis improves the sample complexity bound recently given by Agarwal et al. [4].Preprint. Under review.

Improved Regret Bound and Experience Replay in Regularized Policy Iteration

“…In this section, we show that the estimation error condition in Assumption 5.2 (and thus O( √ T ) regret) can be achieved under similar assumptions as in Abbasi-Yadkori et al (2019) and Wei et al (2020a), which we state next. Assumption 6.1 (Linear value functions).…”

“…Hao et al (2020) improve these results to O(T 2/3 ). More recently, Wei et al (2020a) present three algorithms for average-reward MDPs with linear function approximation. Among these, FOPO achieves O( √ T ) regret but is computationally inefficient, and OLSVI.FH is efficient but obtains O(T 3/4 ) regret.…”

“…With linear value functions, the Politex algorithm of Abbasi-Yadkori et al (2019) achieves O(T 3/4 ) high-probability regret in ergodic MDPs, and the results only scale in the number of features rather than states. Wei et al (2020a) later show an O( √ T ) bound on expected regret for a similar algorithm named MDP-EXP2. In this work, we revisit the analysis of Politex and show that it can be sharpened to O( √ T ) under nearly identical assumptions, resulting in the first O( √ T ) high-probability regret bound for a computationally efficient algorithm in this setting.…”

“…The original work of Even-Dar et al (2009) studied this scheme with known dynamics, tabular representation, and adversarial reward functions. More recent works (Abbasi-Yadkori et al, 2019;Hao et al, 2020;Wei et al, 2020a) have adapted the approach to the case of unknown dynamics, stochastic rewards, and value function approximation. With linear value functions, the Politex algorithm of Abbasi-Yadkori et al (2019) achieves O(T 3/4 ) high-probability regret in ergodic MDPs, and the results only scale in the number of features rather than states.…”

Nearly Minimax Optimal Regret for Learning Infinite-horizon Average-reward MDPs with Linear Function Approximation

“…To overcome the curse of large state space, function approximation has been used to design practically successful algorithms (Singh et al, 1995;Mnih et al, 2015;Bertsekas, 2018). However, most existing studies on learning infinite-horizon average-reward MDPs are limited to tabular MDPs, with only a few exceptions (Abbasi-Yadkori et al, 2019a,b;Hao et al, 2020;Wei et al, 2020). More specifically, Abbasi-Yadkori et al (2019a,b); Hao et al (2020) studied RL with function approximation for infinite-horizon average-reward MDPs under strong assumptions such as uniformly-mixing and uniformly excited feature, and proved sublinear regrets.…”