Psychological foundations of explainability and interpretability in artificial intelligence

Abstract

In this paper, we make the case that interpretability and explainability are distinct requirements for machine learning systems. To make this case, we provide an overview of the literature in experimental psychology pertaining to interpretation (especially of numerical stimuli) and comprehension. We find that interpretation refers to the ability to contextualize a model's output in a manner that relates it to the system's designed functional purpose, and the goals, values, and preferences of end users. In contrast, explanation refers to the ability to accurately describe the mechanism, or implementation, that led to an algorithm's output, often so that the algorithm can be improved in some way. Beyond these definitions, our review shows that humans differ from one another in systematic ways, that affect the extent to which they prefer to make decisions based on detailed explanations versus less precise interpretations. These individual differences, such as personality traits and skills, are associated with their abilities to derive meaningful interpretations from precise explanations of model output. This implies that system output should be tailored to different types of users.

Towards a standard for identifying and managing bias in artificial intelligence

“…This issue becomes particularly relevant when deploying systems for use by subject matter experts, who are less interested in how a system works and more concerned with why a system provided a given output. When system designers do not take these perceptual differences into consideration it can lead to misinterpretation of output, which is especially problematic in high-risk settings [247,248]. Coordinated guidance is necessary to ensure that transparency tools are effectively supporting the professionals who use them and not indirectly contributing to processes that could amplify bias.…”

Quality Models for Artificial Intelligence Systems: Characteristic-Based Approach, Development and Application

“…In the process of analyzing key references related to the principles and characteristics of AI [13][14][15][16][17][18]22], as well as other sources provided in Table 1, the following are identified:…”

“…Among such technologies are the most complex and promising means of artificial intelligence (AI). Evidence of the growing dynamics of the implementation of AI systems (AIS) in various fields, as well as the intensity of development and research, is the rapid increase in the number of publications during 2018-2021 [1], accepted and developed standards and guides EU Commission [2,3], ISO/IEC [4][5][6][7][8][9][10], IEEE [11,12], NIST [13][14][15][16][17][18], OECD [19][20][21], and UNESCO [22].…”

A general-purpose method for applying Explainable AI for Anomaly Detection

“…According to [7] explainability is the model's ability to provide a description of how a model's outcome came to be, and interpretability refers to a human's ability to make sense, or derive meaning, from a given stimulus so that the human can make a decision. Similar to [7], we propose that explainability and interpretability are two distinct ideas.…”