Assessing the Impact of Automated Suggestions on Decision Making: Domain Experts Mediate Model Errors but Take Less Initiative

Abstract

Automated decision support can accelerate tedious tasks as users can focus their attention where it is needed most. However, a key concern is whether users overly trust or cede agency to automation. In this paper, we investigate the effects of introducing automation to annotating clinical texts -a multi-step, error-prone task of identifying clinical concepts (e.g., procedures) in medical notes, and mapping them to labels in a large ontology. We consider two forms of decision aid: recommending which labels to map concepts to, and pre-populating annotation suggestions. Through laboratory studies, we find that 18 clinicians generally build intuition of when to rely on automation and when to exercise their own judgement. However, when presented with fully pre-populated suggestions, these expert users exhibit less agency: accepting improper mentions, and taking less initiative in creating additional annotations. Our findings inform how systems and algorithms should be designed to mitigate the observed issues. CCS CONCEPTS• Human-centered computing → HCI theory, concepts and models; • Applied computing → Health informatics.

AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts

“…In a collaboration, AI can play various roles, such as casual creators that encourage exploration [27] or assistants that compensate for human weaknesses [42,72]. For example, Gero et al [30] showed that generators could serve as cognitive offloading tools so that humans could focus their attention where it is needed most, a core motivation that we also share.…”

How does HCI Understand Human Autonomy and Agency?

“…In some works, for instance, organisations benefited from efficiency and motivational outcomes, while individuals also benefited materially in the form of improved health outcomes [50] financial benefits [155], and lifestyle flexibility [166]. In other cases, the benefits to the user were primarily experiential, while the organisation saw material benefits such as improved work-outcomes, and stronger attachment in games, products and services [e.g., 8,41,104,166]. Finally, in some instances, it was not clear that the user gained benefits that they would have chosen for themselves.…”

Is explainable AI a race against model complexity?

“…For example, the ASSESS MS system used by clinical neurologists to assess multiple sclerosis is part of a wider process involving multiple tools, procedures, and performances of expert judgment [11]. Just as they would seek to understand magnetic resonance imaging (MRI) as part of their process -the strengths, limitations and idiosyncrasies of the tool, and develop the 'professional vision' [12] re-quired to effectively read MRI images through the lenses of technical and medical knowledge -so too did they seek to understand how the predictions of an AI model could be incorporated into the process of diagnosis, a finding replicated in other clinical contexts [13].…”