How Do Users Experience Traceability of AI Systems? Examining Subjective Information Processing Awareness in Automated Insulin Delivery (AID) Systems

Abstract

When interacting with artificial intelligence (AI) in the medical domain, users frequently face automated information processing, which can remain opaque to them. For example, users with diabetes may interact daily with automated insulin delivery (AID). However, effective AID therapy requires traceability of automated decisions for diverse users. Grounded in research on human-automation interaction, we study Subjective Information Processing Awareness (SIPA) as a key construct to research users’ experience of explainable AI. The objective of the present research was to examine how users experience differing levels of traceability of an AI algorithm. We developed a basic AID simulation to create realistic scenarios for an experiment with N = 80, where we examined the effect of three levels of information disclosure on SIPA and performance. Attributes serving as the basis for insulin needs calculation were shown to users, who predicted the AID system’s calculation after over 60 observations. Results showed a difference in SIPA after repeated observations, associated with a general decline of SIPA ratings over time. Supporting scale validity, SIPA was strongly correlated with trust and satisfaction with explanations. The present research indicates that the effect of different levels of information disclosure may need several repetitions before it manifests. Additionally, high levels of information disclosure may lead to a miscalibration between SIPA and performance in predicting the system’s results. The results indicate that for a responsible design of XAI, system designers could utilize prediction tasks in order to calibrate experienced traceability.

Anchoring Autonomy: Understanding Seafarers’ Interaction with Energy Efficiency Decision Support Systems for Route Planning and the Role of Basic Psychological Needs

“…Moreover, as automated systems become more capable and ubiquitous, the criteria for assessing and interacting with them are likely to evolve. Traditional usability and User Experience metrics may soon reach a 'ceiling effect,' necessitating new metrics like BPN-TU or PATS, and other scales from the field of XAI, such as the Subjective Information Processing Awareness Scale (Schrills and Franke 2023). This underscores the need for tools that can capture the new dimensions of interaction, possibly resembling scales from social psychology, as with system trust (Hoff and Bashir 2015).…”