Learning Options for an MDP from Demonstrations

Tamassia, Marco; Zambetta, Fabio; Raffe, William L.; Li, Xiaodong

doi:10.1007/978-3-319-14803-8_18

Cited by 4 publications

(4 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Key areas for future work include (1) relaxing the assumption that controllers are provided; (2) learning better abstractions from even fewer demonstrations by performing active learning to gather more data online; (3) expanding the expressivity of the grammar to learn more sophisticated predicates; and (4) applying the ideas presented in this paper to non-deterministic and/or partially observed planning problems. For (1), the literature on option learning [89,90] provides a starting point; in our setting, it would be necessary to not only learn the initiation sets, policies, and termination conditions of the options [51], but also segment the demonstrations appropriately once options have been learned, since now these demonstrations would not contain actions. For (2), we hope to investigate how relational exploration algorithms [53,91] might be useful as a mechanism for an agent to decide what actions to execute, toward the goal of building better state and action abstractions.…”

Section: Discussionmentioning

confidence: 99%

Inventing Relational State and Action Abstractions for Effective and Efficient Bilevel Planning

Silver¹,

Chitnis²,

Kumar³

et al. 2022

Preprint

View full text Add to dashboard Cite

Effective and efficient planning in continuous state and action spaces is fundamentally hard, even when the transition model is deterministic and known. One way to alleviate this challenge is to perform bilevel planning with abstractions, where a high-level search for abstract plans is used to guide planning in the original transition space. In this paper, we develop a novel framework for learning state and action abstractions that are explicitly optimized for both effective (successful) and efficient (fast) bilevel planning. Given demonstrations of tasks in an environment, our data-efficient approach learns relational, neuro-symbolic abstractions that generalize over object identities and numbers. The symbolic components resemble the STRIPS predicates and operators found in AI planning, and the neural components refine the abstractions into actions that can be executed in the environment. Experimentally, we show across four robotic planning environments that our learned abstractions are able to quickly solve held-out tasks of longer horizons than were seen in the demonstrations, and can even outperform the efficiency of abstractions that we manually specified. We also find that as the planner configuration varies, the learned abstractions adapt accordingly, indicating that our abstraction learning method is both "task-aware" and "planner-aware." Code: https://tinyurl.com/predicators-release

show abstract

Section: Discussionmentioning

confidence: 99%

Inventing Relational State and Action Abstractions for Effective and Efficient Bilevel Planning

Silver¹,

Chitnis²,

Kumar³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Tamassia et al . (2016) suggest a different approach: dynamically selecting state-space abstraction by which different states that share the same abstraction features are considered similar. Sequeira et al .…”

Section: Preliminaries and Backgroundmentioning

confidence: 99%

Leveraging human knowledge in tabular reinforcement learning: a study of human subjects

Rosenfeld

Cohen²,

Taylor³

et al. 2018

The Knowledge Engineering Review

View full text Add to dashboard Cite

Reinforcement Learning (RL) can be extremely effective in solving complex, real-world problems. However, injecting human knowledge into an RL agent may require extensive effort and expertise on the human designer's part. To date, human factors are generally not considered in the development and evaluation of possible RL approaches. In this article, we set out to investigate how different methods for injecting human knowledge are applied, in practice, by human designers of varying levels of knowledge and skill. We perform the first empirical evaluation of several methods, including a newly proposed method named SASS which is based on the notion of similarities in the agent's state-action space. Through this human study, consisting of 51 human participants, we shed new light on the human factors that play a key role in RL. We find that the classical reward shaping technique seems to be the most natural method for most designers, both expert and non-expert, to speed up RL. However, we further find that our proposed method SASS can be effectively and efficiently combined with reward shaping, and provides a beneficial alternative to using only a single speedup method with minimal human designer effort overhead.

show abstract

“…While the above methods consider the expert data at a global scale, our work is concerned with the problem of subgoal modeling, which is often conducted in the form of option-based reasoning (Sutton et al, 1999). For instance, Tamassia et al (2015) proposed a clustering approach based on state distances to find a minimal set of options that can explain the expert behavior. While the method provides a simple alternative to handcrafting options, it does not allow any probabilistic treatment of the data and involves many ad-hoc design choices.…”

Section: Related Workmentioning

confidence: 99%

“…. , |S|} indicates the subgoal location and C ∈ (0, ∞) is some positive constant (compare S ¸imşek et al, 2005;Stolle and Precup, 2002;Tamassia et al, 2015).…”

Section: Revisiting the Bnirl Frameworkmentioning

confidence: 99%

Inverse Reinforcement Learning via Nonparametric Spatio-Temporal Subgoal Modeling

Šošić,

Rueckert,

Peters

et al. 2018

Preprint

View full text Add to dashboard Cite

Advances in the field of inverse reinforcement learning (IRL) have led to sophisticated inference frameworks that relax the original modeling assumption of observing an agent behavior that reflects only a single intention. Instead of learning a global behavioral model, recent IRL methods divide the demonstration data into parts, to account for the fact that different trajectories may correspond to different intentions, e.g., because they were generated by different domain experts. In this work, we go one step further: using the intuitive concept of subgoals, we build upon the premise that even a single trajectory can be explained more efficiently locally within a certain context than globally, enabling a more compact representation of the observed behavior. Based on this assumption, we build an implicit intentional model of the agent's goals to forecast its behavior in unobserved situations. The result is an integrated Bayesian prediction framework that significantly outperforms existing IRL solutions and provides smooth policy estimates consistent with the expert's plan. Most notably, our framework naturally handles situations where the intentions of the agent change over time and classical IRL algorithms fail. In addition, due to its probabilistic nature, the model can be straightforwardly applied in active learning scenarios to guide the demonstration process of the expert.

show abstract

Learning Options for an MDP from Demonstrations

Cited by 4 publications

References 14 publications

Inventing Relational State and Action Abstractions for Effective and Efficient Bilevel Planning

Inventing Relational State and Action Abstractions for Effective and Efficient Bilevel Planning

Leveraging human knowledge in tabular reinforcement learning: a study of human subjects

Inverse Reinforcement Learning via Nonparametric Spatio-Temporal Subgoal Modeling

Contact Info

Product

Resources

About