Interestingness elements for explainable reinforcement learning: Understanding agents' capabilities and limitations

Sequeira, Pedro; Gervasio, Melinda T.

doi:10.1016/j.artint.2020.103367

Cited by 88 publications

(70 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To evaluate participants' ability to differentiate between alternative agents and analyze their strategies, we trained agents that behave qualitatively different. To this end, we modified the reward function used for training (similar approach to that used by Sequeira et al [65]), resulting in three types of agents. As mentioned in section 5, we based all of those reward functions on the default ALE [14] reward function, which measures the increase in in-game score (as described above) between the first and last frame of a state.…”

Section: Methodsmentioning

confidence: 99%

“…We chose to use this approach since it does not make any assumptions about people's reasoning, is simpler computationally and was shown to improve users' understanding of agent behavior. Similar approaches have been developed in parallel [34,65], varying in the specific formulation of the interestingness criteria used to determine which states to include in the summary.…”

Section: Related Workmentioning

confidence: 99%

“…Their results show that summary methods that better match with people's computational models lead to improved action prediction, but that people may use different models in different contexts. Summaries generated by a variety of interestingness criteria were shown to improve people's ability to identify regions of the state space in which an agent spends more time and regions of the state space in which an agent requires additional training [65]. Importance-based summaries (e.g.…”

Section: Related Workmentioning

confidence: 99%

“…The videos we generate from the states chosen by the summary show 30 frames per second. To emphasize that demonstrations show different trajectories, they are separated by a black screen that appears for 1 second (inspired by the fade-out effect used in [65]). To prevent the users from using the in-game score to gauge how good an agent is, we mask the bottom half of the screen with black pixels.…”

Section: Integrating Local and Global Informationmentioning

confidence: 99%

See 3 more Smart Citations

Local and global explanations of agent behavior: Integrating strategy summaries with saliency maps

Huber

Weitz

André

et al. 2021

Artificial Intelligence

View full text Add to dashboard Cite

With advances in reinforcement learning (RL), agents are now being developed in high-stakes application domains such as healthcare and transportation. Explaining the behavior of these agents is challenging, as the environments in which they act have large state spaces, and their decision-making can be affected by delayed rewards, making it difficult to analyze their behavior. To address this problem, several approaches have been developed. Some approaches attempt to convey the global behavior of the agent, describing the actions it takes in different states. Other approaches devised local explanations which provide information regarding the agent's decision-making in a particular state. In this paper, we combine global and local explanation methods, and evaluate their joint and separate contributions, providing (to the best of our knowledge) the first user study of combined local and global explanations for RL agents. Specifically, we augment strategy summaries that extract important trajectories of states from simulations of the agent with saliency maps which show what information the agent attends to. Our results show that the choice of what states to include in the summary (global information) strongly affects people's understanding of agents: participants shown summaries that included important states significantly outperformed participants who were presented with agent behavior in a randomly set of chosen world-states. We find mixed results with respect to augmenting demonstrations with saliency maps (local information), as the addition of saliency maps did not significantly improve performance in most cases. However, we do find some evidence that saliency maps can help users better understand what information the agent relies on in its decision making, suggesting avenues for future work that can further improve explanations of RL agents.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Integrating Local and Global Informationmentioning

confidence: 99%

See 2 more Smart Citations

Local and global explanations of agent behavior: Integrating strategy summaries with saliency maps

Huber

Weitz

André

et al. 2021

Artificial Intelligence

View full text Add to dashboard Cite

show abstract

“…One approach is to visualise an agent's current state and factors which affect the agent's decision making (Iyer et al 2018). An exception is a method which summarises an agent's strategy in a video (Sequeira and Gervasio 2020). In this work, agents do not play optimally and the videos are used to allow the human to assess the capabilities of the agent.…”

Section: Explanations For Human Problem Solving and Sequential Decisimentioning

confidence: 99%

Beneficial and harmful explanatory machine learning

Muggleton

Hocquette

et al. 2021

Mach Learn

View full text Add to dashboard Cite

Given the recent successes of Deep Learning in AI there has been increased interest in the role and need for explanations in machine learned theories. A distinct notion in this context is that of Michie’s definition of ultra-strong machine learning (USML). USML is demonstrated by a measurable increase in human performance of a task following provision to the human of a symbolic machine learned theory for task performance. A recent paper demonstrates the beneficial effect of a machine learned logic theory for a classification task, yet no existing work to our knowledge has examined the potential harmfulness of machine’s involvement for human comprehension during learning. This paper investigates the explanatory effects of a machine learned theory in the context of simple two person games and proposes a framework for identifying the harmfulness of machine explanations based on the Cognitive Science literature. The approach involves a cognitive window consisting of two quantifiable bounds and it is supported by empirical evidence collected from human trials. Our quantitative and qualitative results indicate that human learning aided by a symbolic machine learned theory which satisfies a cognitive window has achieved significantly higher performance than human self learning. Results also demonstrate that human learning aided by a symbolic machine learned theory that fails to satisfy this window leads to significantly worse performance than unaided human learning.

show abstract