The challenge of crafting intelligible intelligence

Abstract

Since Artificial Intelligence (AI) software uses techniques like deep lookahead search and stochastic optimization of huge neural networks to fit mammoth datasets, it often results in complex behavior that is difficult for people to understand. Yet organizations are deploying AI algorithms in many mission-critical settings. To trust their behavior, we must make AI intelligible, either by using inherently interpretable models or by developing new methods for explaining and controlling otherwise overwhelmingly complex decisions using local approximation, vocabulary alignment, and interactive explanation. This paper argues that intelligibility is essential, surveys recent work on building such systems, and highlights key directions for research.

Explanation in artificial intelligence: Insights from the social sciences

“…Given that early work on XAI was to explain symbolic approaches, the authors of such work would have more intuitively seen the link to interaction. Despite this, others in the AI community have recently re-discovered the importance of social interaction for explanation; for example, [186,163], and have noted that this is a problem that requires collaboration with HCI researchers.…”

Trusting Intelligent Machines: Deepening Trust Within Socio-Technical Systems

“…In many cases this is due to the increased ability of machines to work constantly, consistently, at scale, and at speed. In some areas, for example Deep Mind's Alpha Go Zero, the results appear to exceed human ability; some moves are made that are novel and inexplicable to human Goplaying experts and yet are effective, leading to more wins and new insights into the game [16]. This raises the question of the limits of explainability.…”

Explainable Machine Learning for Scientific Insights and Discoveries

“…One motivation behind this is that scientists increasingly adopt ML for optimizing and producing scientific outcomes, where explainability is a prerequisite to ensure the scientific value of the outcome. In this context, research directions such as explainable artificial intelligence (AI) , informed ML [von Rueden et al, 2019], or intelligible intelligence [Weld and Bansal, 2018] have emerged. Though related, the concepts, goals, and motivations vary, and core technical terms are defined in different ways.…”