Block Policy Mirror Descent

Lan, Guanghui; Li, Yan; Zhao, Tao

doi:10.48550/arxiv.2201.05756

Cited by 3 publications

(5 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Until recently, linear convergence of policy gradient method is established in [13,26] for MDPs without any regularization, by proposing an approximate policy mirror descent method. Policy gradient methods with a block update rule, which evaluate and update the policy for only a subset of states, have been also proposed in [14], with similar convergence properties. With softmax policy parameterization, linear convergence of natural policy gradient method is also established in [12] when using adaptive stepsizes.…”

Section: Introductionmentioning

confidence: 99%

“…(1) Convergence rate: Linear convergence of policy gradient methods has been discussed in prior literature [14,12,26], while existing superlinear convergence result often requires strong algorithm-dependent assumptions [21,12]. It is unclear whether policy gradient methods converge superlinearly in general settings.…”

Section: Introductionmentioning

confidence: 99%

“…For more environment details, we refer readers to[14], which adopt the same experiment setup. Note that in Figure1a, we vary the discount factor while keeping other defining parameters of the MDP unchanged.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Homotopic Policy Mirror Descent: Policy Convergence, Implicit Regularization, and Improved Sample Complexity

Li¹,

Zhao²,

Lan³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

We propose the homotopic policy mirror descent (HPMD) method for solving discounted, infinite horizon MDPs with finite state and action space, and study its policy convergence. We report three properties that seem to be new in the literature of policy gradient methods: (1) HPMD exhibits global linear convergence of the value optimality gap, and local superlinear convergence of the policy to the set of optimal policies with order γ −2 . The superlinear convergence of the policy takes effect after no more than O(log(1/∆ * )) number of iterations, where ∆ * is defined via a gap quantity associated with the optimal state-action value function; (2) HPMD also exhibits last-iterate convergence of the policy, with the limiting policy corresponding exactly to the optimal policy with the maximal entropy for every state. No regularization is added to the optimization objective and hence the second observation arises solely as an algorithmic property of the HPMD method. (3) For the stochastic HPMD method, we further demonstrate a better than O(|S||A|/ 2 ) sample complexity for small optimality gap , when assuming a generative model for policy evaluation.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Homotopic Policy Mirror Descent: Policy Convergence, Implicit Regularization, and Improved Sample Complexity

Li¹,

Zhao²,

Lan³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…By carefully solving a sequence of entropy-regularized MDPs, with diminishing regularizations and increasing stepsizes, [21] proposes the first linearly-converging PG method for un-regularized MDPs. This is further simplified in [23,27,42], which drops the regularization while retaining the linear convergence. Beyond the optimality gap, convergence of the policy has been studied in [28].…”

Section: Introductionmentioning

confidence: 99%

Policy Mirror Descent Inherently Explores Action Space

Li¹,

Lan²

2023

Preprint

View full text Add to dashboard Cite

Explicit exploration in the action space was assumed to be indispensable for online policy gradient methods to avoid a drastic degradation in sample complexity, for solving general reinforcement learning problems over finite state and action spaces. In this paper, we establish for the first time an O(1/ǫ 2 ) sample complexity for online policy gradient methods without incorporating any exploration strategies. The essential development consists of two new on-policy evaluation operators and a novel analysis of the stochastic policy mirror descent method (SPMD) [21]. SPMD with the first evaluation operator, called value-based estimation, tailors to the Kullback-Leibler divergence. Provided the Markov chains on the state space of generated policies are uniformly mixing with non-diminishing minimal visitation measure, an O(1/ǫ 2 ) sample complexity is obtained with a linear dependence on the size of the action space. SPMD with the second evaluation operator, namely truncated on-policy Monte Carlo (TOMC), attains an O(HD/ǫ 2 ) sample complexity, where HD mildly depends on the effective horizon and the size of the action space with properly chosen Bregman divergence (e.g., Tsallis divergence). SPMD with TOMC also exhibits stronger convergence properties in that it controls the optimality gap with high probability rather than in expectation. In contrast to explicit exploration, these new policy gradient methods can prevent repeatedly committing to potentially high-risk actions when searching for optimal policies.

show abstract

“…

…”

mentioning

confidence: 99%

On the Convergence of Policy in Unregularized Policy Mirror Descent

Lin¹,

Zhang²

2022

Preprint

View full text Add to dashboard Cite

In this short note, we give the convergence analysis of the policy in recent famous policy mirror descent (PMD) [5,12,8,6,11]. We mainly consider the unregularized setting following [11] with generalized Bregman divergence. The difference is that we directly give the convergence rates of policy under generalized Bregman divergence. Our results are inspired by the convergence of value function in previous works and are an extension study of policy mirror descent. Though some results have already appeared in previous work Khodadadian et al. [4], Li et al. [8], we further discover a large body of Bregman divergences could give finite-step-convergence to an optimal policy, such as the classical Euclidean distance.

show abstract

Block Policy Mirror Descent

Cited by 3 publications

References 17 publications

Homotopic Policy Mirror Descent: Policy Convergence, Implicit Regularization, and Improved Sample Complexity

Homotopic Policy Mirror Descent: Policy Convergence, Implicit Regularization, and Improved Sample Complexity

Policy Mirror Descent Inherently Explores Action Space

On the Convergence of Policy in Unregularized Policy Mirror Descent

Contact Info

Product

Resources

About