Is Temporal Difference Learning Optimal? An Instance-Dependent Analysis

Khamaru, Koulik; Pananjady, Ashwin; Ruan, Feng; Wainwright, Martin J.; Jordan, Michael I.

doi:10.48550/arxiv.2003.07337

Cited by 8 publications

(13 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A number of impactful analysis techniques have been developed for this setting with corresponding minimax bounds (Azar et al, 2013;Sidford et al, 2018;Agarwal et al, 2020;Li et al, 2020). Recently, several instance-dependent results have been shown in the generative model setting Khamaru et al, 2020Khamaru et al, , 2021. Most relevant is the work of which proposes the Bespoke algorithm and achieves a sample complexity of s,a log(1/δ) max{ 2 ,∆(s,a) 2 } , ignoring horizon dependence.…”

Section: Related Workmentioning

confidence: 99%

Beyond No Regret: Instance-Dependent PAC Reinforcement Learning

Wagenmaker¹,

Simchowitz²,

Jamieson³

2021

Preprint

View full text Add to dashboard Cite

The theory of reinforcement learning has focused on two fundamental problems: achieving low regret, and identifying -optimal policies. While a simple reduction allows one to apply a low-regret algorithm to obtain an -optimal policy and achieve the worst-case optimal rate, it is unknown whether low-regret algorithms can obtain the instance-optimal rate for policy identification. We show that this is not possible-there exists a fundamental tradeoff between achieving low regret and identifying an -optimal policy at the instance-optimal rate. Motivated by our negative finding, we propose a new measure of instance-dependent sample complexity for PAC tabular reinforcement learning which explicitly accounts for the attainable state visitation distributions in the underlying MDP. We then propose and analyze a novel, planning-based algorithm which attains this sample complexity-yielding a complexity which scales with the suboptimality gaps and the "reachability" of a state. We show that our algorithm is nearly minimax optimal, and on several examples that our instance-dependent sample complexity offers significant improvements over worst-case bounds.

show abstract

Section: Related Workmentioning

confidence: 99%

Beyond No Regret: Instance-Dependent PAC Reinforcement Learning

Wagenmaker¹,

Simchowitz²,

Jamieson³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Establishing information-theoretic or algorithmic-specific lower bounds on the statistical and computational complexities of RL algorithms -often achieved by constructing hard MDP instances -plays an instrumental role in understanding the bottlenecks of RL algorithms. To give a few examples, Azar et al (2013) established an information-theoretic lower bound on the sample complexity of learning the optimal policy in a generative model, whereas Khamaru et al (2020); Pananjady and Wainwright (2020) developed instance-dependent lower bounds for policy evaluation. Additionally, Agarwal et al (2019) constructed a chain-like MDP whose value function under direct parameterization might contain very flat saddle points under a certain initial state distribution, highlighting the role of distribution mismatch coefficients in policy optimization.…”

Section: Other Related Workmentioning

confidence: 99%

Softmax Policy Gradient Methods Can Take Exponential Time to Converge

Wei

Chi

et al. 2021

Preprint

View full text Add to dashboard Cite

The softmax policy gradient (PG) method, which performs gradient ascent under softmax policy parameterization, is arguably one of the de facto implementations of policy optimization in modern reinforcement learning. For γ-discounted infinite-horizon tabular Markov decision processes (MDPs), remarkable progress has recently been achieved towards establishing global convergence of softmax PG methods in finding a near-optimal policy. However, prior results fall short of delineating clear dependencies of convergence rates on salient parameters such as the cardinality of the state space S and the effective horizon 1 1−γ , both of which could be excessively large. In this paper, we deliver a pessimistic message regarding the iteration complexity of softmax PG methods, despite assuming access to exact gradient computation. Specifically, we demonstrate that softmax PG methods can take exponential time -in terms of |S| and 1 1−γ -to converge, even in the presence of a benign policy initialization and an initial state distribution amenable to exploration. This is accomplished by characterizing the algorithmic dynamics over a carefully-constructed MDP containing only three actions. Our exponential lower bound hints at the necessity of carefully adjusting update rules or enforcing proper regularization in accelerating PG methods.

show abstract

“…The seminal idea of variance reduction was originally proposed to accelerate finite-sum stochastic optimization, e.g., Gower et al (2020); Johnson and Zhang (2013); Nguyen et al (2017). Thereafter, the variance reduction strategy has been imported to RL, which assists in improving the sample efficiency of RL algorithms in multiple contexts, including but not limited to policy evaluation (Du et al, 2017;Khamaru et al, 2020;Wai et al, 2019;Xu et al, 2019), RL with a generative model (Sidford et al, 2018a,b;Wainwright, 2019b), asynchronous Q-learning (Li et al, 2020b), and offline RL (Yin et al, 2021).…”

Section: Related Workmentioning

confidence: 99%

Breaking the Sample Complexity Barrier to Regret-Optimal Model-Free Reinforcement Learning

Li¹,

Shi²,

Chen³

et al. 2021

Preprint

View full text Add to dashboard Cite

Achieving sample efficiency in online episodic reinforcement learning (RL) requires optimally balancing exploration and exploitation. When it comes to a finite-horizon episodic Markov decision process with S states, A actions and horizon length H, substantial progress has been achieved towards characterizing the minimax-optimal regret, which scales on the order of √ H 2 SAT (modulo log factors) with T the total number of samples. While several competing solution paradigms have been proposed to minimize regret, they are either memory-inefficient, or fall short of optimality unless the sample size exceeds an enormous threshold (e.g., S 6 A 4 poly(H) for existing model-free methods).To overcome such a large sample size barrier to efficient RL, we design a novel model-free algorithm, with space complexity O(SAH), that achieves near-optimal regret as soon as the sample size exceeds the order of SA poly(H). In terms of this sample size requirement (also referred to the initial burnin cost), our method improves -by at least a factor of S 5 A 3 -upon any prior memory-efficient algorithm that is asymptotically regret-optimal. Leveraging the recently introduced variance reduction strategy (also called reference-advantage decomposition), the proposed algorithm employs an early-settled reference update rule, with the aid of two Q-learning sequences with upper and lower confidence bounds. The design principle of our early-settled variance reduction method might be of independent interest to other RL settings that involve intricate exploration-exploitation trade-offs.

show abstract

Is Temporal Difference Learning Optimal? An Instance-Dependent Analysis

Cited by 8 publications

References 24 publications

Beyond No Regret: Instance-Dependent PAC Reinforcement Learning

Beyond No Regret: Instance-Dependent PAC Reinforcement Learning

Softmax Policy Gradient Methods Can Take Exponential Time to Converge

Breaking the Sample Complexity Barrier to Regret-Optimal Model-Free Reinforcement Learning

Contact Info

Product

Resources

About