Plan Explanations as Model Reconciliation -- An Empirical Study

Abstract

Recent work in explanation generation for decision making agents has looked at how unexplained behavior of autonomous systems can be understood in terms of differences in the model of the system and the human's understanding of the same, and how the explanation process as a result of this mismatch can be then seen as a process of reconciliation of these models. Existing algorithms in such settings, while having been built on contrastive, selective and social properties of explanations as studied extensively in the psychology literature, have not, to the best of our knowledge, been evaluated in settings with actual humans in the loop. As such, the applicability of such explanations to human-AI and human-robot interactions remains suspect. In this paper, we set out to evaluate these explanation generation algorithms in a series of studies in a mock search and rescue scenario with an internal semi-autonomous robot and an external human commander. We demonstrate to what extent the properties of these algorithms hold as they are evaluated by humans, and how the dynamics of trust between the human and the robot evolve during the process of these interactions.From the perspective of planning and decision making, the notion of explanations of the deliberative process of an AI-based system was first explored extensively in the context of expert systems [24]. Similar techniques have been looked at for explanations in case based planning systems [16,28] and in interactive planning [26]

A Survey of Mental Modeling Techniques in Human–Robot Teaming

“…These communications can be verbal (explicit) or nonverbal (implicit), as seen in Section "Mental Model Methodologies." For explicit models, the following qualities have been found to be positively correlated with trust and teamwork: taskrelated communications, contrastive explanations expressing model divergence, and user and context-dependent information (such as providing technical information to an expert, and accessible information to a lay-user) [77][78][79]. For implicit models, such as those aimed at plan legibility and explicability, self-reported understanding of a robotic agents' behavior or goal is a common evaluation metric.…”

Explaining in Time

“…Unless otherwise clarified, it carries implicit commitments to what will happen as the robot continues to operate in this context. Chakraborti et al, for example, have couched explanation as "plan reconciliation" in order for human-robot teams to share an understanding through explanation that enables and presupposes future collaboration [Chakraborti et al 2019].…”

“…Explanations, from that vantage, can be a means toward improving human-robot performance. It has been couched, for example, as "plan reconciliation," wherein human-robot teams might share an understanding through explanation that enables and presupposes future collaboration [Chakraborti et al 2019]. Hayes et al see the value of explanation in achieving robot controller transparency [Hayes and Shah 2017].…”

The Emerging Landscape of Explainable Automated Planning & Decision Making

“…Such explanations are inherently social in being able to explicitly capture the effect of expectations in the explanation process. In user studies conducted in [12], it was shown that participants were indeed able to identify the correct τ based on an explanation. Note that, in the model reconciliation framework, the mental model is just a version of the decision making problem at hand which the agent believes the user is operating under.…”