Learning Stochastic Shortest Path with Linear Function Approximation

Abstract

We study the stochastic shortest path (SSP) problem in reinforcement learning with linear function approximation, where the transition kernel is represented as a linear mixture of unknown models. We call this class of SSP problems the linear mixture SSP. We propose a novel algorithm for learning the linear mixture SSP, which can attain a O(dB 1.5 K/c min ) regret. Here K is the number of episodes, d is the dimension of the feature mapping in the mixture model, B bounds the expected cumulative cost of the optimal policy, and c min > 0 is the lower bound of the cost function. Our algorithm also applies to the case when c min = 0, where a O(K 2/3 ) regret is guaranteed. To the best of our knowledge, this is the first algorithm with a sublinear regret guarantee for learning linear mixture SSP. In complement to the regret upper bounds, we also prove a lower bound of Ω(dB √ K), which nearly matches our upper bound.

Improved No-Regret Algorithms for Stochastic Shortest Path with Linear MDP

“…Importantly, this bound is horizon-free in the sense that it has no polynomial dependency on T ⋆ or 1 cmin even in the lower order terms. Moreover, this almost matches the best known lower bound Ω(dB ⋆ √ K) from (Min et al, 2021).…”

“…There is some gap between our result above and the existing lower bound Ω(dB ⋆ √ K) for this problem (Min et al, 2021). In particular, the dependency on T ⋆ inherited from the H dependency in Lemma 2 is most likely unnecessary.…”

“…Ignoring the lower order term, our bound is (potentially) suboptimal only in terms of the d-dependency compared to the lower bound Ω(dB ⋆ √ K) from (Min et al, 2021). We note again that this is the first horizon-free regret bound for linear SSP: it does not have any polynomial dependency on T ⋆ or 1 cmin even in the lower order terms.…”

“…Our result is obtained by developing a simpler and improved analysis for the finite-horizon approximation of (Cohen et al, 2021) with a smaller approximation error, which might be of independent interest. On the other hand, using variance-aware confidence sets in a global optimization problem, our second algorithm is computationally inefficient but achieves the first "horizon-free" regret bound Õ(d 3.5 B ⋆ √ K) with no polynomial dependency on T ⋆ or 1/c min , almost matching the Ω(dB ⋆ √ K) lower bound from (Min et al, 2021).…”

“…Here, d is the dimension of the feature space, B ⋆ is an upper bound on the expected costs of the optimal policy, and c min is the minimum cost across all state-action pairs. Later, Min et al (2021) study a related but different SSP problem defined over a linear mixture MDP and achieve a Õ(dB 1.5 ⋆ K/c min ) regret bound. Despite leveraging the advances from both the finite-horizon and infinite-horizon settings, results above are still far from optimal in terms of regret guarantee or computational efficiency, demonstrating the unique challenge of SSP problems.…”