User Satisfaction with an AI System for Chest X-Ray Analysis Implemented in a Hospital’s Emergency Setting

Abstract

The acceptance of artificial intelligence (AI) systems by health professionals is crucial to obtain a positive impact on the diagnosis pathway. We evaluated user satisfaction with an AI system for the automated detection of findings in chest x-rays, after five months of use at the Emergency Department. We collected quantitative and qualitative data to analyze the main aspects of user satisfaction, following the Technology Acceptance Model. We selected the intended users of the system as study participants: radiology residents and emergency physicians. We found that both groups of users shared a high satisfaction with the system’s ease of use, while their perception of output quality (i.e., diagnostic performance) differed notably. The perceived usefulness of the application yielded positive evaluations, focusing on its utility to confirm that no findings were omitted, and also presenting distinct patterns across the two groups of users. Our results highlight the importance of clearly differentiating the intended users of AI applications in clinical workflows, to enable the design of specific modifications that better suit their particular needs. This study confirmed that measuring user acceptance and recognizing the perception that professionals have of the AI system after daily use can provide important insights for future implementations.

Automation in Contemporary Clinical Information Systems: a Survey of AI in Healthcare Settings

“…Although a staged approach to implementation and evaluation was evident in many studies (e.g., [48,66]), only three tracked actual use of systems by clinicians [28,29,75]. Evaluation of user experience was mostly confined to assessing satisfaction via surveys.…”

“…Nine studies examined systems for a variety of clinical areas in hospital and outpatient radiology departments. Taking a theory driven approach, Rabinovich et al [28] used the Technology Acceptance Model to evaluate user satisfaction and actual use of an assistive system for chest x-ray interpretation in an Argentinian emergency department (ED) over 5-months. The system was used for 15% of studies (n=1,186), with an average of eight accesses per day.…”

Attitudes towards artificial intelligence in emergency medicine

“…7 Comparison to previous qualitative studies (emergency clinicians) Previous qualitative interview-based studies of emergency clinicians' attitudes towards AI have had a smaller number of participants and mostly focused on attitudes towards specific AI-based tools, such as detecting pathology in chest X-rays, diagnosing aortic dissection, or predicting 30-day mortality. [30][31][32][33] In these studies, attitudes towards AI were generally positive, with clinicians viewing AI as a tool to supplement clinical expertise and help inexperienced clinicians. [30][31][32][33] However, there were also concerns that AI could bias clinical decisions, and that inexperienced clinicians could become over-reliant on such tools.…”

“…[30][31][32][33] In these studies, attitudes towards AI were generally positive, with clinicians viewing AI as a tool to supplement clinical expertise and help inexperienced clinicians. [30][31][32][33] However, there were also concerns that AI could bias clinical decisions, and that inexperienced clinicians could become over-reliant on such tools. [31][32][33] Other concerns included the trustworthiness of AI systems, alarm fatigue, the medicolegal risk of documenting AI-based predictions, the impact on clinician autonomy, and the multiple human factors that could be overlooked by AI.…”

“…Previous qualitative interview‐based studies of emergency clinicians' attitudes towards AI have had a smaller number of participants and mostly focused on attitudes towards specific AI‐based tools, such as detecting pathology in chest X‐rays, diagnosing aortic dissection, or predicting 30‐day mortality 30–33 . In these studies, attitudes towards AI were generally positive, with clinicians viewing AI as a tool to supplement clinical expertise and help inexperienced clinicians 30–33 . However, there were also concerns that AI could bias clinical decisions, and that inexperienced clinicians could become over‐reliant on such tools 31–33 .…”