Learning to Control in Metric Space with Optimal Regret

Ni, Chengzhuo; Yang, Lin F.; Wang, Mengdi

doi:10.1109/allerton.2019.8919864

Cited by 7 publications

(5 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While the optimal lower bound of the error of RL algorithms in the tabular setting has been established in [2,4], there are much fewer results about lower bounds of RL with function approximation. [26] proves an optimal lower bound for Lipschitz function approximation. [27] shows that even when the value function, policy function, reward function, and transition probability can be approximated by a linear function, it is still possible that solving the RL problem requires samples exponentially depending on the horizon.…”

Section: Our Contributionsmentioning

confidence: 99%

Perturbational Complexity by Distribution Mismatch: A Systematic Analysis of Reinforcement Learning in Reproducing Kernel Hilbert Space

Long¹,

Han²

2022

JML

View full text Add to dashboard Cite

Most existing theoretical analysis of reinforcement learning (RL) is limited to the tabular setting or linear models due to the difficulty in dealing with function approximation in high dimensional space with an uncertain environment. This work offers a fresh perspective into this challenge by analyzing RL in a general reproducing kernel Hilbert space (RKHS). We consider a family of Markov decision processes M of which the reward functions lie in the unit ball of an RKHS and transition probabilities lie in a given arbitrary set. We define a quantity called perturbational complexity by distribution mismatch ∆ M (ǫ) to characterize the complexity of the admissible state-action distribution space in response to a perturbation in the RKHS with scale ǫ. We show that ∆ M (ǫ) gives both the lower bound of the error of all possible algorithms and the upper bound of two specific algorithms (fitted reward and fitted Q-iteration) for the RL problem. Hence, the decay of ∆ M (ǫ) with respect to ǫ measures the difficulty of the RL problem on M. We further provide some concrete examples and discuss whether ∆ M (ǫ) decays fast or not in these examples. As a byproduct, we show that when the reward functions lie in a high dimensional RKHS, even if the transition probability is known and the action space is finite, it is still possible for RL problems to suffer from the curse of dimensionality.

show abstract

Section: Our Contributionsmentioning

confidence: 99%

Perturbational Complexity by Distribution Mismatch: A Systematic Analysis of Reinforcement Learning in Reproducing Kernel Hilbert Space

Long¹,

Han²

2022

JML

View full text Add to dashboard Cite

show abstract

“…In more structured environments, an agent can frequently take advantage of metric-based learning (Kakade, Kearns, and Langford 2003). Recent theoretical results have bounded cumulative regret by assuming Lipschitz continuity of either the optimal Q-function or the transition function, in both deterministic (Ni, Yang, and Wang 2019) and stochastic (Pazis and Parr 2013;Touati, Taiga, and Bellemare 2020) domains. However, due to their restrictive assumptions these methods may be computationally impractical to apply as the dimensionality of the learning domain increases.…”

Section: Related Workmentioning

confidence: 99%

“…As stated earlier, we would like knownness to quantify similarity of the state-action to previously observed state-actions. We assume X is endowed with a metric, d(x 1 , x 2 ), that quantifies such similarity, as is common in many works in continuous RL (Ni, Yang, and Wang 2019;Asadi, Misra, and Littman 2018). We define knownness as a function of the distance to the closest state-action which the agent has observed:…”

Section: Optimistic Initialization In Continuous Mdpsmentioning

confidence: 99%

Optimistic Initialization for Exploration in Continuous Control

Lobel

Gottesman

Allen

et al. 2022

AAAI

View full text Add to dashboard Cite

Optimistic initialization underpins many theoretically sound exploration schemes in tabular domains; however, in the deep function approximation setting, optimism can quickly disappear if initialized naively. We propose a framework for more effectively incorporating optimistic initialization into reinforcement learning for continuous control. Our approach uses metric information about the state-action space to estimate which transitions are still unexplored, and explicitly maintains the initial Q-value optimism for the corresponding state-action pairs. We also develop methods for efficiently approximating these training objectives, and for incorporating domain knowledge into the optimistic envelope to improve sample efficiency. We empirically evaluate these approaches on a variety of hard exploration problems in continuous control, where our method outperforms existing exploration techniques.

show abstract

“…Related Literature While the optimal lower bound of the error of RL algorithms in the tabular setting has been established in [3,4], there are much fewer results about lower bounds of RL with function approximation. [31] proves an optimal lower bound for Lipschitz function approximation. [14] shows that even when the value function, policy function, reward function, and transition probability can be approximated by a linear function, it is still possible that solving the RL problem requires samples exponentially depending on the horizon.…”

Section: Our Contributionmentioning

confidence: 99%

Perturbational Complexity by Distribution Mismatch: A Systematic Analysis of Reinforcement Learning in Reproducing Kernel Hilbert Space

Long¹,

Han²

2021

Preprint

View full text Add to dashboard Cite

Most existing theoretical analysis of reinforcement learning (RL) is limited to the tabular setting or linear models due to the difficulty in dealing with function approximation in high dimensional space with an uncertain environment. This work offers a fresh perspective into this challenge by analyzing RL in a general reproducing kernel Hilbert space (RKHS). We consider a family of Markov decision processes M of which the reward functions lie in the unit ball of an RKHS and transition probabilities lie in a given arbitrary set. We define a quantity called perturbational complexity by distribution mismatch ∆M(ǫ) to characterize the complexity of the admissible state-action distribution space in response to a perturbation in the RKHS with scale ǫ. We show that ∆M(ǫ) gives both the lower bound of the error of all possible algorithms and the upper bound of two specific algorithms (fitted reward and fitted Q-iteration) for the RL problem. Hence, the decay of ∆M(ǫ) with respect to ǫ measures the difficulty of the RL problem on M. We further provide some concrete examples and discuss whether ∆M(ǫ) decays fast or not in these examples. As a byproduct, we show that when the reward functions lie in a high dimensional RKHS, even if the transition probability is known and the action space is finite, it is still possible for RL problems to suffer from the curse of dimensionality.

show abstract

Learning to Control in Metric Space with Optimal Regret

Cited by 7 publications

References 7 publications

Perturbational Complexity by Distribution Mismatch: A Systematic Analysis of Reinforcement Learning in Reproducing Kernel Hilbert Space

Perturbational Complexity by Distribution Mismatch: A Systematic Analysis of Reinforcement Learning in Reproducing Kernel Hilbert Space

Optimistic Initialization for Exploration in Continuous Control

Perturbational Complexity by Distribution Mismatch: A Systematic Analysis of Reinforcement Learning in Reproducing Kernel Hilbert Space

Contact Info

Product

Resources

About