A Tool for Evaluating Medication Alerting Systems: Development and Initial Assessment

Abstract

Background It is well known that recommendations from electronic medication alerts are seldom accepted or acted on by users. Key factors affecting the effectiveness of medication alerts include system usability and alert design. Thus, human factors principles that apply knowledge of human capabilities and limitations are increasingly used in the design of health technology to improve the usability of systems. Objective This study aims to evaluate a newly developed evidence-based self-assessment tool that allows the valid and reliable evaluation of computerized medication alerting systems. This tool was developed to be used by hospital staff with detailed knowledge of their hospital’s computerized provider order entry system and alerts to identify and address potential system deficiencies. In this initial assessment, we aim to determine whether the items in the tool can measure compliance of medication alerting systems with human factors principles of design, the tool can be consistently used by multiple users to assess the same system, and the items are easy to understand and perceived to be useful for assessing medication alerting systems. Methods The Tool for Evaluating Medication Alerting Systems (TEMAS) was developed based on human factors design principles and consisted of 66 items. In total, 18 staff members recruited across 6 hospitals used the TEMAS to assess their medication alerting systems. Data collected from participant assessments were used to evaluate the validity, reliability, and usability of the TEMAS. Validity was assessed by comparing the results of the TEMAS with those of prior in-house evaluations. Reliability was measured using Krippendorff α to determine agreement among assessors. A 7-item survey was used to determine usability. Results The participants reported mostly negative (n=8) and neutral (n=7) perceptions of alerts in their medication alerting system. However, the validity of the TEMAS could not be directly tested, as participants were unaware of any results from prior in-house evaluations. The reliability of the TEMAS, as measured by Krippendorff α, was low to moderate (range 0.26-0.46); however, participant feedback suggests that individuals’ knowledge of the system varied according to their professional background. In terms of usability, 61% (11/18) of participants reported that the TEMAS items were generally easy to understand; however, participants suggested the revision of 22 items to improve clarity. Conclusions This initial assessment of the TEMAS allowed the identification of its components that required modification to improve usability and usefulness. It also revealed that for the TEMAS to be effective in facilitating a comprehensive assessment of a medication alerting system, it should be completed by a multidisciplinary team of hospital staff from both clinical and technical backgrounds to maximize their knowledge of systems.

Comparison of the validity, perceived usefulness, and usability of I-MeDeSA and TEMAS, two tools to evaluate alert system usability

“…The original versions of I-MeDeSA [12] and TEMAS [13] were translated into French by a human factors researcher with experience in medication alert systems. Two human factors specialists and one clinician also with expertise in health technology, all fluent in French and in English, and with expertise in decision support, rated their level of agreement with the translation of each item.…”

“…With respect to medication alert systems, two paper-based evaluation tools have been developed to assist vendors and stakeholders in determining whether an alert system complies with usability design principles. These are the Instrument for evaluating human factors principles in medicationrelated decision support alerts (I-MeDeSA) [12] and the Tool for evaluating medication alerting systems (TEMAS) [13], as described in Table 1.…”

Comparison of the validity, perceived usefulness and usability of I-MeDeSA and TEMAS, two tools to evaluate alert system usability: a study protocol

“…To address the shortcomings identified in I-MeDeSA, Zheng et al 16 developed a tool to assess the usability of medication alerts using a list of evidence-based usability principles specific to medication alert systems: the TEMAS. The designers' objective was to develop a tool that could be used by hospital staff with extensive knowledge of the prescribing software and the alert system (eg, computerised physician order entry (CPOE) pharmacist) to evaluate their system.…”

“…A preliminary evaluation of TEMAS was recently performed 16 and this led to the identification of a number of confusing items. As a result, 22 items were revised to reduce ambiguity.…”

Cognitive task analysis of clinicians’ drug–drug interaction management during patient care and implications for alert design

“…Recent studies focus on DDI alert usability. Useful recommendations9 10 and alert evaluation tools,11 12 such as I-MedDeSA13 and TEMAS,14 have been published based on human factors engineering design principles. Newer DDI alert designs include optional Internet links, such as to detailed DDI monographs, for decision-making 10 15.…”