Design and validation of a questionnaire to evaluate the usability of computerized critical care information systems

Abstract

The implementation of computerized critical care information systems (CCIS) can improve the quality of clinical care and staff satisfaction, but also holds risks of disrupting the workflow with consecutive negative impacts. The usability of CCIS is one of the key factors determining their benefits and weaknesses. However, no tailored instrument exists to measure the usability of such systems. Therefore, the aim of this study was to design and validate a questionnaire that measures the usability of CCIS. Following a mixed-method design approach, we developed a questionnaire comprising two evaluation models to assess the usability of CCIS: (1) the task-specific model rates the usability individually for several tasks which CCIS could support and which we derived by analyzing work processes in the ICU; (2) the characteristic-specific model rates the different aspects of the usability, as defined by the international standard "ergonomics of human-system interaction". We tested validity and reliability of the digital version of the questionnaire in a sample population. In the sample population of 535 participants both usability evaluation models showed a strong correlation with the overall rating of the system (multiple correlation coefficients ≥0.80) as well as a very high internal consistency (Cronbach's alpha ≥0.93). The novel questionnaire is a valid and reliable instrument to measure the usability of CCIS and can be used to study the influence of the usability on their implementation benefits and weaknesses.

Investigation of the Usability of Computerized Critical Care Information Systems in Germany

“…Validity of the rating models is based on content validity (structured development of the models following a holistic approach) as well as previously published analyses of criterion validity (conformity between different rating models with multiple correlation coefficients !0.8) and construct validity (intercorrelation analysis of the elements of the model rating the technical usability characteristics). 33 The reliability of the rating models has been demonstrated in a previously published analysis of the internal consistency and was shown to have Cronbach's a values of ! 0.93.…”

“…Extensive validation has shown that the questionnaire actually measures the usability of CCIS. 33 Because the questionnaire co-registers the known confounders of subjective usability ratings and their influences are included in our analyses, our results should be robust. However, a questionnaire study cannot achieve the level of internal validity that individual user interviews, observed user tests, or expert testing could achieve for usability assessment.…”

“…To investigate the different aspects of CCIS usability, we used a previously published and validated questionnaire. 33 The questionnaire contains three rating models of CCIS usability: (1) an overall rating of the system, (2) a model rating technical usability, and (3) a model rating task-specific usability.…”

“…0.93. 33 Along with the three rating models of CCIS usability, the questionnaire contains questions about user and workplace characteristics that might influence ratings. The conceptual model of the questionnaire is presented in Figure 1.…”

“…We performed a web-based nationwide survey among ICU physicians and nurses using a specially designed questionnaire that allows to measure the usability of CCIS separately within a variety of work processes in an ICU. 33…”