Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergence

Zhan, Wenhao; Cen, Shicong; Huang, Baihe; Chen, Yuxin; Lee, Jason D.; Chi, Yuejie

doi:10.48550/arxiv.2105.11066

Cited by 15 publications

(35 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is also worth mentioning that by extending similar arguments, the local superlinear convergence can also be established for the approximate policy mirror descent (APMD) method developed in [13,26].…”

Section: Local Superlinear Convergence and Implicit Regularizationmentioning

confidence: 97%

“…In a nutshell, the HPMD method can be considered as a simplification of the approximate policy mirror descent (APMD) method proposed in [13] and extended in [26], by dropping the need for evaluating the perturbed stateaction value function Q π k τ k , defined by…”

Section: Homotopic Policy Mirror Descent: Linear Convergencementioning

confidence: 99%

“…Linear convergence with constant stepsize is shown in [4] for solving entropy-regularized MDPs, when applying the natural policy gradient method [11] with softmax policy parameterization. Until recently, linear convergence of policy gradient method is established in [13,26] for MDPs without any regularization, by proposing an approximate policy mirror descent method. Policy gradient methods with a block update rule, which evaluate and update the policy for only a subset of states, have been also proposed in [14], with similar convergence properties.…”

Section: Introductionmentioning

confidence: 99%

“…(1) Convergence rate: Linear convergence of policy gradient methods has been discussed in prior literature [14,12,26], while existing superlinear convergence result often requires strong algorithm-dependent assumptions [21,12]. It is unclear whether policy gradient methods converge superlinearly in general settings.…”

Section: Introductionmentioning

confidence: 99%

“…(2) Convergence of the policy: It is not clear whether policy gradient methods exhibits last-iterate convergence of the policy, as existing developments mostly establish convergence in optimality gap. To the best of our knowledge, it seems that only asymptotic subsequence convergence to an optimal policy can be claimed based on the value convergence [13,26,1]. More importantly, even when policy gradient methods exhibit last-iterate convergence, it is also unclear whether one can characterize the particular policy that the method converges to, as the number of optimal policies can be infinite.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Homotopic Policy Mirror Descent: Policy Convergence, Implicit Regularization, and Improved Sample Complexity

Li¹,

Zhao²,

Lan³

2022

Preprint

View full text Add to dashboard Cite

We propose the homotopic policy mirror descent (HPMD) method for solving discounted, infinite horizon MDPs with finite state and action space, and study its policy convergence. We report three properties that seem to be new in the literature of policy gradient methods: (1) HPMD exhibits global linear convergence of the value optimality gap, and local superlinear convergence of the policy to the set of optimal policies with order γ −2 . The superlinear convergence of the policy takes effect after no more than O(log(1/∆ * )) number of iterations, where ∆ * is defined via a gap quantity associated with the optimal state-action value function; (2) HPMD also exhibits last-iterate convergence of the policy, with the limiting policy corresponding exactly to the optimal policy with the maximal entropy for every state. No regularization is added to the optimization objective and hence the second observation arises solely as an algorithmic property of the HPMD method. (3) For the stochastic HPMD method, we further demonstrate a better than O(|S||A|/ 2 ) sample complexity for small optimality gap , when assuming a generative model for policy evaluation.

show abstract

Section: Local Superlinear Convergence and Implicit Regularizationmentioning

confidence: 97%

Section: Homotopic Policy Mirror Descent: Linear Convergencementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Homotopic Policy Mirror Descent: Policy Convergence, Implicit Regularization, and Improved Sample Complexity

Li¹,

Zhao²,

Lan³

2022

Preprint

View full text Add to dashboard Cite

show abstract

Softmax policy gradient methods can take exponential time to converge

Wei

Chi

et al. 2023

Math. Program.

View full text Add to dashboard Cite

The softmax policy gradient (PG) method, which performs gradient ascent under softmax policy parameterization, is arguably one of the de facto implementations of policy optimization in modern reinforcement learning. For $$\gamma $$ γ -discounted infinite-horizon tabular Markov decision processes (MDPs), remarkable progress has recently been achieved towards establishing global convergence of softmax PG methods in finding a near-optimal policy. However, prior results fall short of delineating clear dependencies of convergence rates on salient parameters such as the cardinality of the state space $${\mathcal {S}}$$ S and the effective horizon $$\frac{1}{1-\gamma }$$ 1 1 - γ , both of which could be excessively large. In this paper, we deliver a pessimistic message regarding the iteration complexity of softmax PG methods, despite assuming access to exact gradient computation. Specifically, we demonstrate that the softmax PG method with stepsize $$\eta $$ η can take $$\begin{aligned} \frac{1}{\eta } |{\mathcal {S}}|^{2^{\Omega \big (\frac{1}{1-\gamma }\big )}} ~\text {iterations} \end{aligned}$$ 1 η | S | 2 Ω ( 1 1 - γ ) iterations to converge, even in the presence of a benign policy initialization and an initial state distribution amenable to exploration (so that the distribution mismatch coefficient is not exceedingly large). This is accomplished by characterizing the algorithmic dynamics over a carefully-constructed MDP containing only three actions. Our exponential lower bound hints at the necessity of carefully adjusting update rules or enforcing proper regularization in accelerating PG methods.

show abstract

Policy gradient primal-dual mirror descent for constrained MDPs with large state spaces

Ding

Jovanović

2022

2022 IEEE 61st Conference on Decision and Control (CDC)

View full text Add to dashboard Cite

We study the problem of computing an optimal policy of an infinite-horizon discounted constrained Markov decision process (constrained MDP). Despite the popularity of Lagrangian-based policy search methods used in practice, the oscillation of policy iterates in these methods has not been fully understood, bringing out issues such as violation of constraints and sensitivity to hyper-parameters. To fill this gap, we employ the Lagrangian method to cast a constrained MDP into a constrained saddle-point problem in which max/min players correspond to primal/dual variables, respectively, and develop two single-time-scale policy-based primal-dual algorithms with non-asymptotic convergence of their policy iterates to an optimal constrained policy. Specifically, we first propose a regularized policy gradient primal-dual (RPG-PD) method that updates the policy using an entropy-regularized policy gradient, and the dual via a quadratic-regularized gradient ascent, simultaneously. We prove that the policy primal-dual iterates of RPG-PD converge to a regularized saddle point with a sublinear rate, while the policy iterates converge sublinearly to an optimal constrained policy. We further instantiate RPG-PD in large state or action spaces by including function approximation in policy parametrization, and establish similar sublinear last-iterate policy convergence. Second, we propose an optimistic policy gradient primal-dual (OPG-PD) method that employs the optimistic gradient method to update primal/dual variables, simultaneously. We prove that the policy primal-dual iterates of OPG-PD converge to a saddle point that contains an optimal constrained policy, with a linear rate. To the best of our knowledge, this work appears to be the first non-asymptotic policy last-iterate convergence result for single-time-scale algorithms in constrained MDPs. We further exhibit the merits and effectiveness of our methods in computational experiments.

show abstract

Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergence

Cited by 15 publications

References 34 publications

Homotopic Policy Mirror Descent: Policy Convergence, Implicit Regularization, and Improved Sample Complexity

Homotopic Policy Mirror Descent: Policy Convergence, Implicit Regularization, and Improved Sample Complexity

Softmax policy gradient methods can take exponential time to converge

Policy gradient primal-dual mirror descent for constrained MDPs with large state spaces

Contact Info

Product

Resources

About