Explicability? Legibility? Predictability? Transparency? Privacy? Security? The Emerging Landscape of Interpretable Agent Behavior

Abstract

There has been significant interest of late in generating behavior of agents that is interpretable to the human (observer) in the loop. However, the work in this area has typically lacked coherence on the topic, with proposed solutions for “explicable”, “legible”, “predictable” and “transparent” planning with overlapping, and sometimes conflicting, semantics all aimed at some notion of understanding what intentions the observer will ascribe to an agent by observing its behavior. This is also true for the recent works on “security” and “privacy” of plans which are also trying to answer the same question, but from the opposite point of view – i.e. when the agent is trying to hide instead of reveal its intentions. This paper attempts to provide a workable taxonomy of relevant concepts in this exciting and emerging field of inquiry.

Explaining Preference-Driven Schedules: The EXPRES Framework

“…Explanations are essential for humans to understand the outputs and decisions made by AI systems (Core et al 2006;Miller 2019). There exist many works that provide explanations for different AI use cases ranging from automated planning (Fox, Long, and Magazzeni 2017;Chakraborti et al 2019) to machine learning (Carvalho, Pereira, and Cardoso 2019) or deep learning (Samek, Wiegand, and Müller 2017). Explanations are also crucial in multi-agent environments where some extra challenges arise, such as privacy preservation or fairness (Kraus et al 2020).…”

A Network Flow Interpretation of Robust Goal Legibility in Path Finding

“…In the AI and robotics communities, there has been growing interest in interpretable agent behavior in the past few years (Dragan, Lee, and Srinivasa 2013;Langley et al 2017;Gunning and Aha 2019;Chakraborti et al 2019;Sreedharan et al 2021), stemming from the consideration that rarely, if ever, agents act in isolation from humans. Synthesizing interpretable behavior facilitates smoother Human-AI interaction and also supports trust in autonomy (Bhatt, Ravikumar, and Moura 2019).…”

“…Synthesizing interpretable behavior facilitates smoother Human-AI interaction and also supports trust in autonomy (Bhatt, Ravikumar, and Moura 2019). Interpretability has been studied along three main dimensions, legibility, explicability and predictability (Chakraborti et al 2019), but, lately, some effort has been made to connect and integrate these concepts in unified frameworks (Sreedharan et al 2021; Miura and Zilberstein 2021). We will limit our discussion to legibility and the most relevant related work.…”

The Mirror Agent Model: A Bayesian Architecture for Interpretable Agent Behavior

“…Generally speaking, each method regularizes a specific part of the agent's behavior to match an observer's expectations, therefore reducing the ambiguity that the agent's intention have in the observer model (see Figure 1). Depending on the specific technique the observer model is designed to be interested in different part of intentions such as goals, future plans, or underlying beliefs [6], and thus each interpretability technique regularizes corresponding parts of the agent's intentional model.…”

“…In addition, the agent has an estimate of the intentional model about itself that is possessed by the observer, P H R , which provides the agent information on how its intention is being understood. P H R is a second order theory of mind focused on the observer's inferences about the agent [6]. In this context, the behavior of the agent is therefore a balance between three types of behavior: optimal behavior, interpretable behavior and explanations, all together having the general objective of fulfilling the agent's intention while keeping |P R − P H R |, the distance between the intentional models, low.…”