Approximate Exploration through State Abstraction

Taïga, Adrien Ali; Courville, Aaron; Bellemare, Marc G.

doi:10.48550/arxiv.1808.09819

Cited by 1 publication

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Then, by examining the worst-case estimate of this model, they guarantee safety on continuous environments. We also refer the reader to a rich line of work outside of the safety literature that has studied similarity metrics in RL, in order to improve computation time of planning and sample complexity of exploration [19][20][21][22][23] (see Appendix B.6 for more discussion).…”

Section: Related Workmentioning

confidence: 99%

“…These state aggregations allow for more efficient planning and exploration [22]. Other work has used pseudo-counts to learn approximate state aggregations [23]. The reason we do not use state-aggregation methods for transferring dynamics knowledge is that we want to include environments where similarities cannot easily partition the state space, such as situations where the similarity between two states is proportional to their distance.…”

Section: B6 Mdp Metrics and Our Analogy Functionmentioning

confidence: 99%

See 1 more Smart Citation

Provably Safe PAC-MDP Exploration Using Analogies

Roderick¹,

Nagarajan²,

Kolter³

2020

Preprint

View full text Add to dashboard Cite

A key challenge in applying reinforcement learning to safety-critical domains is understanding how to balance exploration (needed to attain good performance on the task) with safety (needed to avoid catastrophic failure). Although a growing line of work in reinforcement learning has investigated this area of "safe exploration," most existing techniques either 1) do not guarantee safety during the actual exploration process; and/or 2) limit the problem to a priori known and/or deterministic transition dynamics with strong smoothness assumptions. Addressing this gap, we propose Analogous Safe-state Exploration (ASE), an algorithm for provably safe exploration in MDPs with unknown, stochastic dynamics. Our method exploits analogies between state-action pairs to safely learn a near-optimal policy in a PAC-MDP sense. Additionally, ASE also guides exploration towards the most task-relevant states, which empirically results in significant improvements in terms of sample efficiency, when compared to existing methods. Source code for the experiments is available at https://github.com/locuslab/ase.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: B6 Mdp Metrics and Our Analogy Functionmentioning

confidence: 99%