System Usability Scale Benchmarking for Digital Health Apps: Meta-analysis

Hyzy, Maciej Marek; Bond, Raymond; Mulvenna, Maurice; Bai, Lu; Dix, Alan; Leigh, Simon; Hunt, Sophie

doi:10.2196/37290

Cited by 119 publications

(105 citation statements)

References 32 publications

(37 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Participants preferred the chatbot's interface, reporting higher usability and significantly higher likelihood to recommend it as a data collection tool. The reported SUS scores for the chatbot (69.7) was higher than the average industry benchmark score of 68 (19,36). The NPS score of the chatbot (NPS = 24) was higher than the online form (NPS = 13).…”

Section: Main Findingsmentioning

confidence: 78%

Virtual conversational agents versus online forms: Patient experience and preferences for health data collection

Soni¹,

Ivanova²,

Wilczewski³

et al. 2022

Front. Digit. Health

View full text Add to dashboard Cite

ObjectiveVirtual conversational agents, or chatbots, have emerged as a novel approach to health data collection. However, research on patient perceptions of chatbots in comparison to traditional online forms is sparse. This study aimed to compare and assess the experience of completing a health assessment using a chatbot vs. an online form.MethodsA counterbalanced, within-subject experimental design was used with participants recruited via Amazon Mechanical Turk (mTurk). Participants completed a standardized health assessment using a chatbot (i.e., Dokbot) and an online form (i.e., REDCap), each followed by usability and experience questionnaires. To address poor data quality and preserve integrity of mTurk responses, we employed a thorough data cleaning process informed by previous literature. Quantitative (descriptive and inferential statistics) and qualitative (thematic analysis and complex coding query) approaches were used for analysis.ResultsA total of 391 participants were recruited, 185 of whom were excluded, resulting in a final sample size of 206 individuals. Most participants (69.9%) preferred the chatbot over the online form. Average Net Promoter Score was higher for the chatbot (NPS = 24) than the online form (NPS = 13) at a statistically significant level. System Usability Scale scores were also higher for the chatbot (i.e. 69.7 vs. 67.7), but this difference was not statistically significant. The chatbot took longer to complete but was perceived as conversational, interactive, and intuitive. The online form received favorable comments for its familiar survey-like interface.ConclusionOur findings demonstrate that a chatbot provided superior engagement, intuitiveness, and interactivity despite increased completion time compared to online forms. Knowledge of patient preferences and barriers will inform future design and development of recommendations and best practice for chatbots for healthcare data collection.

show abstract

Section: Main Findingsmentioning

confidence: 78%

Virtual conversational agents versus online forms: Patient experience and preferences for health data collection

Soni¹,

Ivanova²,

Wilczewski³

et al. 2022

Front. Digit. Health

View full text Add to dashboard Cite

show abstract

“…The SUS tool has been defined as technology-agnostic, suitable for all type of user interfaces, and Hyzy at al. [ 53 ] demonstrated that the use of the SUS could be reliable in case of mHealth solutions. A simplified version of the SUS was proposed by Holden [ 54 ] for cognitively impaired and older adults that could not understand all the SUS statements properly.…”

Section: Discussionmentioning

confidence: 99%

Design and Evaluation of Personalized Services to Foster Active Aging: The Experience of Technology Pre-Validation in Italian Pilots

Lorusso

Mosmondor

Grguric

et al. 2023

Sensors

View full text Add to dashboard Cite

Assistive devices could promote independent living and support the active and healthy aging of an older population; however, several factors can badly influence the long-term use of new technologies. In this context, this paper presents a two-step methodology called “pre-validation” that aims to identify the factors that can bias the use of new services, thus minimizing the risk of an unsuccessful longer trial. The proposed pre-validation methodology is composed of two main phases that aim to assess the usability and the reliability of the technology assessed in a laboratory environment and the usability, acceptability, user experience, and reliability of the technology in real environments. The tested services include the socialization scenario, in which older adults are better connected to the community via technological solutions (i.e., socialization applications), and the monitoring scenario, which allows for the introduction of timely interventions (technologies involved include environmental monitoring sensors, a telepresence robot, wearable sensors, and a personalized dashboard). The obtained results underline an acceptable usability level (average System Usability Scale score > 65) for the tested technologies (i.e., socialization applications and a telepresence robot). Phase Two also underlines the good acceptability, user experience, and usability of the tested services. The statistical analysis underlines a correlation between the stress related to the use of technology, digital skills, and intention of use, among other factors. Qualitative feedback also remarks on a correlation between older adults with low digital skills and an anxiety about using technology. Positive correlation indexes were highlighted between the trust and usability scores. Eventually, future long-term trials with assistive technology should rely on motivated caregivers, be founded on a strong recruitment process, and should reassure older adults—especially the ones with low digital literacy—about the use of technology by proposing personalized training and mentoring, if necessary, to increase the trust.

show abstract

“…There are several possible explanations for these findings. Although the SUS score distribution has been used for benchmarking a vast array of new technologies and more recently shown to be suitable for benchmarking mHealth or digital health apps (excluding apps just for self-monitoring physical activity), it is possible that the benchmark score could be subject to reporting bias in the literature, that is, the file drawer problem, where researchers may be more likely to report only higher usability [ 34 , 53 ]. Low mHealth usability has been found in previous studies in vulnerable, underserved populations based on other usability scales and metrics [ 54 , 55 ]; however, to the best of our knowledge, this is the first time SUS scores have been reported for a digital health app with 2 separate interfaces for 2 types of end users, CMs and AYA living with HIV participants.…”

Section: Discussionmentioning

confidence: 99%

“…The SUS has been validated for use with mobile and web apps and scores range from 0 to 100, with higher scores indicating higher perceived usability [ 33 ]. A score >68 is considered above the industry average [ 34 ].…”

Section: Methodsmentioning

confidence: 99%

A Digital Platform to Support HIV Case Management for Youth and Young Adults: Mixed Methods Feasibility Study

Fee¹,

Fuller²,

Guss³

et al. 2022

JMIR Form Res

View full text Add to dashboard Cite

Background Advances in medical treatments in recent years have contributed to an overall decline in HIV-related opportunistic infections and deaths in youth; however, mortality and morbidity rates in perinatally and nonperinatally infected adolescents and young adults (AYA) living with HIV remain relatively high today. Objective The goal of this project was to assess the use, utility, and cost-effectiveness of PlusCare, a digital app for HIV case management in AYA living with HIV. The app supports routine case management tasks, such as scheduling follow-up visits, sharing documents for review and signature, laboratory test results, and between-visit communications (eg, encouraging messages). Methods We conducted a single-group mixed methods pre-post study with HIV case management programs in 2 large urban hospitals in the Boston metro area. Case management staff (case managers [CMs], N=20) and AYA living with HIV participants (N=45) took part in the study with access to PlusCare for up to 15 and 12 months, respectively. Results The CMs and AYA living with HIV reported mean System Usability Scale scores of 51 (SD 7.9) and 63 (SD 10.6), respectively. Although marginally significant, total charges billed at 1 of the 2 sites compared with the 12 months before app use (including emergency, inpatient, and outpatient charges) decreased by 41% (P=.046). We also observed slight increases in AYA living with HIV self-reported self-efficacy in chronic disease management and quality of life (Health-Related Quality of Life-4) from baseline to the 12-month follow-up (P=.02 and P=.03, respectively) and increased self-efficacy from the 6- to 12-month follow-up (P=.02). There was no significant change in HIV viral suppression, appointment adherence, or medication adherence in this small-sample pilot study. Conclusions Although perceived usability was low, qualitative feedback from CMs and use patterns suggested that direct messaging and timely, remote, and secure sharing of laboratory results and documents (including electronic signatures) between CMs and AYA living with HIV can be particularly useful and have potential value in supporting care coordination and promoting patient self-efficacy and quality of life. Trial Registration ClinicalTrials.gov NCT03758066; https://clinicaltrials.gov/ct2/show/NCT03758066

show abstract

System Usability Scale Benchmarking for Digital Health Apps: Meta-analysis

Cited by 119 publications

References 32 publications

Virtual conversational agents versus online forms: Patient experience and preferences for health data collection

Virtual conversational agents versus online forms: Patient experience and preferences for health data collection

Design and Evaluation of Personalized Services to Foster Active Aging: The Experience of Technology Pre-Validation in Italian Pilots

A Digital Platform to Support HIV Case Management for Youth and Young Adults: Mixed Methods Feasibility Study

Contact Info

Product

Resources

About