Refined Regret for Adversarial MDPs with Linear Function Approximation

Abstract

We consider learning in an adversarial Markov Decision Process (MDP) where the loss functions can change arbitrarily over K episodes and the state space can be arbitrarily large. We assume that the Q-function of any policy is linear in some known features, that is, a linear function approximation exists. The best existing regret upper bound for this setting (Luo et al., 2021b) is of order O(K 2/3 ) (omitting all other dependencies), given access to a simulator. This paper provides two algorithms that improve the regret to O( √ K) in the same setting. Our first algorithm makes use of a refined analysis of the Follow-the-Regularized-Leader (FTRL) algorithm with the logbarrier regularizer. This analysis allows the loss estimators to be arbitrarily negative and might be of independent interest. Our second algorithm develops a magnitude-reduced loss estimator, further removing the polynomial dependency on the number of actions in the first algorithm and leading to the optimal regret bound (up to logarithmic terms and dependency on the horizon). Moreover, we also extend the first algorithm to simulator-free linear MDPs, which achieves O(K 8/9 ) regret and greatly improves over the best existing bound O(K 14/15 ). This algorithm relies on a better alternative to the Matrix Geometric Resampling procedure by Neu and Olkhovskaya (2020), which could again be of independent interest.

Improved Regret Bounds for Linear Adversarial MDPs via Linear Optimization

“…Similar to previous works on linear adversarial MDP [Luo et al, 2021a,b;Sherman et al, 2023;Dai et al, 2023] that discuss the cases of whether a transition simulator is available, we define the simulator that may help in the following assumption. Note that this simulator is weaker than Luo et al [2021a,b]; Dai et al [2023] because their simulator could generate a next state given any state-action pair.…”

“…We could vary this construction procedure by further putting a finite action covering over the action space to deal with the infinite action space setting. More details can be found in Appendix C. Meanwhile, Neu and Olkhovskaya [2021] requires both state and action spaces to be finite and Luo et al [2021a,b]; Dai et al [2023]; Sherman et al [2023] can only deal with finite action space.…”

“…1 Our required simulator is defined in Assumption 2. Notice that Dai et al [2023]; Luo et al [2021a,b] adopt a stronger simulator that returns the next state s ′ when given any state action pair (s, a), while Sherman et al [2023]; Neu and Olkhovskaya [2021] and this paper only need the simulator to return a trajectory when given a policy. 2 Our exploratory assumption is introduced in Assumption 1.…”

“…With the same exploratory assumption as Neu and Olkhovskaya [2021], they show a regret bound O( √ K) with a simulator and O(K 6/7 ) otherwise. Very recent two works [Dai et al, 2023;Sherman et al, 2023] further generalize the setting by removing the exploratory assumption. These two works independently provide O(K 8/9 ) and O(K 6/7 ) regret for this setting when no simulator is available.…”

“…Early works have proposed algorithms when the transition function is known. [Neu and Olkhovskaya, 2021] Several recent works explore the problem without a known transition function and derive policy optimization algorithms with the state-of-the-art regret of Õ K 6/7 [Luo et al, 2021a;Dai et al, 2023;Sherman et al, 2023]. While the optimal regret in tabular MDPs is of order Õ K 1/2 [Jin et al, 2020a], the regret upper bounds available for linear adversarial MDPs seem to admit a large room for improvement.…”