Psychometric Evaluation of the EMO and the SUS in the Context of a Large-Sample Unmoderated Usability Study

Lewis, James R.; Brown, Joshua; Mayes, Daniel K.

doi:10.1080/10447318.2015.1064665

Cited by 34 publications

(27 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our study of the IUS revealed a factor structure that was nearly identical to a SUS study from 2009 [17] with exception of item 6 ("I think there is too much inconsistency in [Intervention]), which we removed. However, our results differed from subsequent studies [21][22][23][24][25][26]. The moderate correlation between the subscales indicates that the measure can be used as a total scale score, as well as decomposed into Usable and Learnable subscales.…”

Section: Discussioncontrasting

confidence: 99%

“…Although the instrument was originally considered unidimensional, Lewis & Sauro [17] assessed the SUS factor structure using 2648 questionnaires assessing products such as computer hardware, software, and websites. They identified an eight-item "Usable" subscale and a two-item "Learnable" subscale, a finding which provoked several replication studies, [21][22][23][24][25] including one analyzing over 9,000 surveys. [26] Unfortunately, these studies did not replicate the initial findings.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Assessing the Usability of Complex Psychosocial Interventions: The Intervention Usability Scale

Lyon

Pullmann

Jacobson

et al. 2020

Preprint

View full text Add to dashboard Cite

Background Usability – the extent to which an intervention can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction – is a key determinant of implementation success. However, usability is rarely assessed in implementation research and no instruments have been developed to measure the design quality of complex health interventions, such as the evidence-based psychosocial interventions that characterize the majority of effective practices in mental and behavioral health services. This study evaluated the structural validity of the Intervention Usability Scale (IUS), an adapted version of the well-established System Usability Scale for digital technologies, when measuring the usability of complex health interventions. Prior studies of the original System Usability Scale have found both one- and two-factor solutions, both of which were examined in the current study of the IUS. Methods A survey was administered to 205 healthcare professionals working at 11 primary care sites. Surveys collected demographic information, including each participant’s professional role (i.e., medical provider, mental/behavioral health provider, pharmacist), and IUS ratings for one of six common evidence-based psychosocial interventions (e.g., cognitive behavioral therapy, motivational interviewing) that they reported using most regularly. Factor analyses replicated the procedures used in prior research on the System Usability Scale, and a sensitivity analysis using analyses of variance compared IUS scores across different groups of respondents and interventions assessed. Results Analyses indicated that a two-factor solution (with “usable” and “learnable” subscales) in which one item was removed best fit the data. This solution accounted for 52.6% of the variance observed. Inter-item reliabilities for the total score, usable subscale, and learnable subscale were α = .83, α = .82, and α = .63, respectively. Resulting scores indicated that usability ranged from below acceptable standards to good, depending on the intervention. On average, behavioral health providers found the interventions to be more usable that other types of healthcare providers. Conclusions The current study provides evidence for a two-factor IUS structure consistent with some prior research, as well as acceptable reliability and sensitivity to role and intervention. Future directions for implementation research evaluating the usability of complex health interventions are discussed.

show abstract

Section: Discussioncontrasting

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Assessing the Usability of Complex Psychosocial Interventions: The Intervention Usability Scale

Lyon

Pullmann

Jacobson

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…However, a large variety of product experience was also reported in studies that found a one-factor model (e.g. ) [24,25]. Another explanation may be related to the complexity of the product; it can be assumed that the Learnability factor has more weight in a context that requires more learning (e.g.…”

Section: Discussionmentioning

confidence: 99%

“…Since then, studies replicated inconsistent findings pointing towards this two-factor model (e.g. ) [23] and the one-factor model as well [24,25]. More recent research showed that two-factor structure possibly depends on the amount of experience that users have with a given product [18].…”

Section: Psychometric Properties and Factor Structure Of The Susmentioning

confidence: 97%

Dimensionality of the system usability scale among professionals using internet-based interventions for depression: a confirmatory factor analysis

et al. 2020

View full text Add to dashboard Cite

Background: The System Usability Scale (SUS) is used to measure usability of internet-based Cognitive Behavioural Therapy (iCBT). However, whether the SUS is a valid instrument to measure usability in this context is unclear. The aim of this study is to assess the factor structure of the SUS, measuring usability of iCBT for depression in a sample of professionals. In addition, the psychometric properties (reliability, convergent validity) of the SUS were tested. Methods: A sample of 242 professionals using iCBT for depression from 6 European countries completed the SUS. Confirmatory Factor Analysis (CFA) was conducted to test whether a one-factor, two-factor, tone-model or bi-direct model would fit the data best. Reliability was assessed using complementary statistical indices (e.g. omega). To assess convergent validity, the SUS total score was correlated with an adapted Client Satisfaction Questionnaire (CSQ-3). Results: CFA supported the one-factor, two-factor and tone-model, but the bi-factor model fitted the data best (Comparative Fit Index = 0.992, Tucker Lewis Index = 0.985, Root Mean Square Error of Approximation = 0.055, Standardized Root Mean Square Residual = 0.042 (respectively χ 2 diff (9) = 69.82, p < 0.001; χ 2 diff (8) = 33.04, p < 0.001). Reliability of the SUS was good (ω = 0.91). The total SUS score correlated moderately with the CSQ-3 (CSQ1 r s = .49, p < 0.001; CSQ2 r s = .46, p < 0.001; CSQ3 r s = .38, p < 0.001), indicating convergent validity. Conclusions: Although the SUS seems to have a multidimensional structure, the best model showed that the total sumscore of the SUS appears to be a valid and interpretable measure to assess the usability of internet-based interventions when used by professionals in mental healthcare.

show abstract

“…Although the instrument was originally considered unidimensional, Sauro and Lewis (2009) evaluated the SUS factor structure using 2648 questionnaires assessing products such as computer hardware, software, and websites. They identi ed an eight-item "Usable" subscale and a two-item "Learnable" subscale, a nding which provoked several replication studies (Kortum & Sorber, 2015;Lewis et al, 2013;Lewis, Brown, et al, 2015;Lewis, Utesch, et al, 2015;Sauro & Lewis, 2011), including one analyzing over 9,000 surveys (Lewis & Sauro, 2017). These studies did not replicate the initial ndings.…”

Section: Human-centered Design (Hcd) and Usabilitymentioning

confidence: 97%

Assessing the Usability of Complex Psychosocial Interventions: The Intervention Usability Scale

Lyon

Pullmann

Jacobson

et al. 2020

Preprint

View full text Add to dashboard Cite

Background. Usability – the extent to which an intervention can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction – is a key determinant of implementation success. However, few instruments have been developed to measure the design quality of complex health interventions (i.e., those with several interacting components). This study evaluated the structural validity of the Intervention Usability Scale (IUS), an adapted version of the well-established System Usability Scale (SUS) for digital technologies, to measure the usability of a leading complex psychosocial intervention, Motivational Interviewing (MI), for behavioral health service delivery in primary care. Prior SUS studies have found both one- and two-factor solutions, both of which were examined in the current study of the IUS.Method. A survey administered to 136 medical professionals from 11 primary care sites collected demographic information and IUS ratings for MI, the evidence-based psychosocial intervention that primary care providers reported using most often for behavioral health service delivery. Factor analyses replicated procedures used in prior research on the SUS.Results. Analyses indicated that a two-factor solution (with “usable” and “learnable” subscales) best fit the data, accounting for 54.1% of the variance. Inter-item reliabilities for the total score, usable subscale, and learnable subscale were α = .83, α = .84, and α = .67, respectively.Conclusions. This study provides evidence for a two-factor IUS structure consistent with some prior research, as well as acceptable reliability. Implications for implementation research evaluating the usability of complex health interventions are discussed, including the potential for future comparisons across multiple interventions and provider types, as well as the use of the IUS to evaluate the relationship between usability and implementation outcomes such as feasibility.

show abstract

Psychometric Evaluation of the EMO and the SUS in the Context of a Large-Sample Unmoderated Usability Study

Cited by 34 publications

References 13 publications

Assessing the Usability of Complex Psychosocial Interventions: The Intervention Usability Scale

Assessing the Usability of Complex Psychosocial Interventions: The Intervention Usability Scale

Dimensionality of the system usability scale among professionals using internet-based interventions for depression: a confirmatory factor analysis

Assessing the Usability of Complex Psychosocial Interventions: The Intervention Usability Scale

Contact Info

Product

Resources

About