Evaluating the Usability and Quality of a Clinical Mobile Application for Assisting Physicians in Head CT Scan Ordering: A Think Aloud and Mobile Apps Rating Scale (MARS) Study (Preprint)

Abstract

Background: Among the numerous factors contributing to healthcare providers' (HCPs) engagement with mobile apps (apps), including user characteristics (dexterity, anatomy, and attitude) and mobile features (screen and button size), usability and quality of apps were introduced as the most influential factors.Objective: Therefore, this study aims to investigate the usability and quality of the Head CT Scan Appropriateness Criteria mobile application (HAC app) for physicians' CT scan ordering.Methods: Our study design was primarily based on methodological triangulation by utilizing mixed methods research involving quantitative analysis of the Mobile Apps Rating Scale (MARS) for quality assessment, quantitative and qualitative Think Aloud (TA) usability testing, and debriefing across three phases. Sixteen medical interns participated in quality assessment and testing usability characteristics, including efficiency, effectiveness, learnability, error, and satisfaction with the HAC app. Results:The effectiveness of the HAC app was deemed satisfactory, with a rating of 96.9%. MARS assessment scale also indicated the overall favorable (82 out of 100) quality score of the HAC app. Scoring four MARS subscales, "Information" (73.37 out of 100) and "Engagement" (73.48 out of 100) had the lowest scores. Analysis of the items in each MARS subscale revealed that in the "Engagement" subscale, the lowest score of the HAC app was "customization" (63.6 out of 100). In the "Functionality" subscale, the HAC app's lowest value was "Performance" (67.4 out of 100). Qualitative TA usability testing of the HAC app found eight notable usability issues also highlighted in the MARS quality assessment. Conclusions:Evaluating the quality and usability of mobile apps using a mixed methods approach provides valuable information about the functionality and disadvantages of the mobile apps and complements each other. It is highly recommended to embrace a more holistic and mixed methods strategy when evaluating mobile apps, as relying solely on one method proves imperfect to reflect trustworthy and reliable information regarding the usability and quality of apps.