Bridging the Gap: Providing Post-Hoc Symbolic Explanations for Sequential Decision-Making Problems with Inscrutable Representations

Abstract

As more and more complex AI systems are introduced into our day-to-day lives, it becomes important that everyday users can work and interact with such systems with relative ease. Orchestrating such interactions require the system to be capable of providing explanations and rationale for its decisions and be able to field queries about alternative decisions. A significant hurdle to allowing for such explanatory dialogue could be the mismatch between the complex representations that the systems use to reason about the task and the terms in which the user may be viewing the task. This paper introduces methods that can be leveraged to provide contrastive explanations in terms of user-specified concepts for deterministic sequential decision-making settings where the system dynamics may be best represented in terms of black box simulators. We do this by assuming that system dynamics can at least be partly captured in terms of symbolic planning models, and we provide explanations in terms of these models. We implement this method using a simulator for a popular Atari game (Montezuma's Revenge) and perform user studies to verify whether people would find explanations generated in this form useful.

Symbol Guided Hindsight Priors for Reward Learning from Human Preferences

“…Our results show, on a Gridworld task, PRIOR is better able to recover rewards that are negative everywhere and decrease further from the goal then the previous state of the PEBBLE [1]. Expanding upon our promising results, we found similar performance benefits of PRIOR over PEBBLE on a variant of Montezuma's Revenge Level 1 [4] and our future work includes extensive evaluation of PRIOR on more complex domains and preferences.…”

“…Finally, we compute both priors over an abstract state representation interpretable to the human-teacher to further boost the agent's reward recovery performance. Humans use symbolic structures when selecting preferences [4], [5] and using the vocabulary of such a symbolic space improves reward attribution to states. The priors use the given symbolic vocabulary and are incorporated into reward function learn-ing as soft constraints [8] on the reward-learning objective as detailed in the next sections.…”

“…Hence we assume a mapping function M : O → P(S), where P(x) represents the power set of x. The mapping function can be elicited from the human in the loop or extracted from domain descriptions as described in [4]. Once the mapping function is available, PRIOR can use the symbols to better interpret the meaning of the preference labels, because the labels are given by the human who tends to accept explanations and reason in symbolic terms [5] and thus priors on the reward values over states built over the symbolic abstraction becomes very useful especially when such priors are being used as soft-constraints.…”

Human‐aware AI —A foundational framework for human–AI interaction

“…Such methods have been employed both within the context of sequential decision-making problems (cf. Sreedharan et al 2020) and single-shot decision-making problems (Kim et al 2018).…”

A Mental-Model Centric Landscape of Human-AI Symbiosis

“…The former corresponds to cases when the behavior (implicit update) or explanation (explicit update) is personalized to the human model and preferences. Examples include providing faithful and customized explanations of a black box classifier that accounts for the fidelity to the original model as well as user interest [30], explanations expressed in the human's vocabulary [50], or choosing explanation based on the human's preference to the type of information [62].…”