Learning Models of Sequential Decision-Making with Partial Specification of Agent Behavior

Abstract

Artificial agents that interact with other (human or artificial) agents require models in order to reason about those other agents’ behavior. In addition to the predictive utility of these models, maintaining a model that is aligned with an agent’s true generative model of behavior is critical for effective human-agent interaction. In applications wherein observations and partial specification of the agent’s behavior are available, achieving model alignment is challenging for a variety of reasons. For one, the agent’s decision factors are often not completely known; further, prior approaches that rely upon observations of agents’ behavior alone can fail to recover the true model, since multiple models can explain observed behavior equally well. To achieve better model alignment, we provide a novel approach capable of learning aligned models that conform to partial knowledge of the agent’s behavior. Central to our approach are a factored model of behavior (AMM), along with Bayesian nonparametric priors, and an inference approach capable of incorporating partial specifications as constraints for model learning. We evaluate our approach in experiments and demonstrate improvements in metrics of model alignment.

Towards an AI Coach to Infer Team Mental Model Alignment in Healthcare

“…For instance, the proposed approach assumes accurate specification of team members' policies and complete observablity of their actions, both of which might be difficult to meet in practice. Hence, we are exploring learning-based approaches to arrive at team policies in presence of latent states [20], [21]. To address the challenges associated with state and action observability, the development of an AI Coach would be greatly enhanced by nuanced surgical tool detection and people tracking methodology.…”

Supervised Bayesian specification inference from demonstrations

“…Konidaris et al (2012), Niekum et al (2015), and Ranchod et al (2015) frame the IRL problem in a semi-Markov setting. Further Unhelkar and Shah (2019) proposed agent Markov models (AMM), a hierarchical approach that models the demonstrator’s policy as piecewise Markov with discrete control modes inferred using a non-parametric prior. These approaches utilize reward functions or policies to represent the task specification implicitly.…”

Models and Algorithms for Human-Aware Task Planning with Integrated Theory of Mind

“…The literature investigated how to create a reasonable model of humans and how to obtain task knowledge, e.g., [22]. Hierarchical models consist of layered abstractions and are considered suitable or close to human intuitions.…”