What Did My AI Learn? How Data Scientists Make Sense of Model Behavior

Abstract

Data scientists require rich mental models of how AI systems behave to effectively train, debug, and work with them. Despite the prevalence of AI analysis tools, there is no general theory describing how people make sense of what their models have learned. We frame this process as a form of sensemaking and derive a framework describing how data scientists develop mental models of AI behavior. To evaluate the framework, we show how existing AI analysis tools fit into this sensemaking process and use it to design AIFinnity , a system for analyzing image-and-text models. Lastly, we explored how data scientists use a tool developed with the framework through a think-aloud study with 10 data scientists tasked with using AIFinnity to pick an image captioning model. We found that AIFinnity ’s sensemaking workflow reflected participants’ mental processes and enabled them to discover and validate diverse AI behaviors.

ScatterShot: Interactive In-context Example Curation for Text Transformation

“…Detecting key phrases in demonstrations. While key phrase extraction in general may require domain knowledge [8,42,65], for text transformation we can leverage the signal present in the relationships between input and output, i.e., in which parts of the input are perturbed or retained. For example, "today" is retained in the output of both "Took a photo today. "…”

Zeno: An Interactive Framework for Behavioral Evaluation of Machine Learning

“…To detect and mitigate these important issues, the ML community uses more fine-grained evaluation approaches, often termed behavioral evaluation [10,47]. Inspired by requirements engineering in software engineering, behavioral evaluation focuses on defining and testing the capabilities of an ML system, its expected behavior on a specification of requirements [45,60].…”

“…There are numerous ML evaluation systems for discovering, validating, and tracking model behaviors [10,47]. The tools use techniques such as visualizations and data transformations to discover behaviors like fairness concerns and edge cases.…”

“…Enumerating what behaviors a model should have or what types of errors it could produce requires collaboration between stakeholders such as ML engineers, designers, and domain experts [40,54]. Behavioral evaluation is also a continuous, iterative process, as practitioners update their models to fix limitations or add features while ensuring that new failures are not introduced [10].…”

ESCAPE: Countering Systematic Errors from Machine’s Blind Spots via Interactive Visual Analysis

“…Unlike existing work, our study proposes an interactive workflow of exploring concepts for the purpose of inspecting systematic errors and spurious concept associations behind them. Similar to [11], our human-in-the-loop workflow aims to promote the sensemaking of practitioners specifically in the problem of systematic errors where they can iteratively work on subsetting, contrasting patterns in instances, and hypothesizing spurious associations.…”