Objective This article reports results from a systematic literature review related to the evaluation of data visualizations and visual analytics technologies within the health informatics domain. The review aims to (1) characterize the variety of evaluation methods used within the health informatics community and (2) identify best practices. Methods A systematic literature review was conducted following PRISMA guidelines. PubMed searches were conducted in February 2017 using search terms representing key concepts of interest: health care settings, visualization, and evaluation. References were also screened for eligibility. Data were extracted from included studies and analyzed using a PICOS framework: Participants, Interventions, Comparators, Outcomes, and Study Design. Results After screening, 76 publications met the review criteria. Publications varied across all PICOS dimensions. The most common audience was healthcare providers (n = 43), and the most common data gathering methods were direct observation (n = 30) and surveys (n = 27). About half of the publications focused on static, concentrated views of data with visuals (n = 36). Evaluations were heterogeneous regarding setting and measurements used. Discussion When evaluating data visualizations and visual analytics technologies, a variety of approaches have been used. Usability measures were used most often in early (prototype) implementations, whereas clinical outcomes were most common in evaluations of operationally-deployed systems. These findings suggest opportunities for both (1) expanding evaluation practices, and (2) innovation with respect to evaluation methods for data visualizations and visual analytics technologies across health settings. Conclusion Evaluation approaches are varied. New studies should adopt commonly reported metrics, context-appropriate study designs, and phased evaluation strategies.

IMPORTANCE Hospitalized children are at increased risk of influenza-related complications, yet influenza vaccine coverage remains low among this group. Evidence-based strategies about vaccination of vulnerable children during all health care visits are especially important during the COVID-19 pandemic. OBJECTIVE To design and evaluate a clinical decision support (CDS) strategy to increase the proportion of eligible hospitalized children who receive a seasonal influenza vaccine prior to inpatient discharge. DESIGN, SETTING, AND PARTICIPANTS This quality improvement study was conducted among children eligible for the seasonal influenza vaccine who were hospitalized in a tertiary pediatric health system providing care to more than half a million patients annually in 3 hospitals. The study used a sequential crossover design from control to intervention and compared hospitalizations in the intervention group (2019-2020 season with the use of an intervention order set) with concurrent controls (2019-2020 season without use of an intervention order set) and historical controls (2018-2019 season with use of an order set that underwent intervention during the 2019-2020 season). INTERVENTIONS A CDS intervention was developed through a user-centered design process, including (1) placing a default influenza vaccine order into admission order sets for eligible patients, (2) a script to offer the vaccine using a presumptive strategy, and (3) just-in-time education for clinicians addressing vaccine eligibility in the influenza order group with links to further reference material. The intervention was rolled out in a stepwise fashion during the 2019-2020 influenza season. MAIN OUTCOMES AND MEASURES Proportion of eligible hospitalizations in which 1 or more influenza vaccines were administered prior to discharge. RESULTS Among 17 740 hospitalizations (9295 boys [52%]), the mean (SD) age was 8.0 (6.0) years, and the patients were predominantly Black (n = 8943 [50%]) or White (n = 7559 [43%]) and mostly had public insurance (n = 11 274 [64%]). There were 10 997 hospitalizations eligible for the influenza vaccine in the 2019-2020 season. Of these, 5449 (50%) were in the intervention group, and 5548 (50%) were concurrent controls. There were 6743 eligible hospitalizations in 2018-2019 that served as historical controls. Vaccine administration rates were 31% (n = 1676) in the intervention group, 19% (n = 1051) in concurrent controls, and 14% (n = 912) in historical controls (P < .001). In adjusted analyses, the odds of receiving the influenza vaccine were 3.25 (95% CI, 2.94-3.59) times higher in the intervention group and 1.28 (95% CI, 1.15-1.42) times higher in concurrent controls than in historical controls. (continued) Key Points Question Is a clinical decision support (CDS) strategy associated with improved influenza vaccination rates before discharge among eligible hospitalized children? Findings In this quality improvement study, the combinination of a defaultchecked influenza vaccine order in admission order sets for eligible p...

Abstract Communication for non-medication order (CNMO) is a type of free text communication order providers use for asynchronous communication about patient care. The objective of this study was to understand the extent to which non-medication orders are being used for medication-related communication. We analyzed a sample of 26 524 CNMOs placed in 6 hospitals. A total of 42% of non-medication orders contained medication information. There was large variation in the usage of CNMOs across hospitals, provider settings, and provider types. The use of CNMOs for communicating medication-related information may result in delayed or missed medications, receiving medications that should have been discontinued, or important clinical decision being made based on inaccurate information. Future studies should quantify the implications of these data entry patterns on actual medication error rates and resultant safety issues.

A radiologist's search pattern can directly influence patient management. A missed finding is a missed opportunity for intervention. Multiple studies have attempted to describe and quantify search patterns but have mainly focused on chest radiographs and chest CTs. Here, we describe and quantify the visual search patterns of 17 radiologists as they scroll through 6 CTs of the abdomen and pelvis. Search pattern tracings varied among individuals and remained relatively consistent per individual between cases. Attendings and trainees had similar eye metric statistics with respect to time to first fixation (TTFF), number of fixations in the region of interest (ROI), fixation duration in ROI, mean saccadic amplitude, or total number of fixations. Attendings had fewer numbers of fixations per second versus trainees (p < 0.001), suggesting efficiency due to expertise. In those cases that were accurately interpreted, TTFF was shorter (p = 0.04), the number of fixations per second and number of fixations in ROI were higher (p = 0.04, p = 0.02, respectively), and fixation duration in ROI was increased (p = 0.02). We subsequently categorized radiologists as Bscanners^or Bdrillers^by both qualitative and quantitative methods and found no differences in accuracy with most radiologists being categorized as Bdrillers.^This study describes visual search patterns of radiologists in interpretation of CTs of the abdomen and pelvis to better approach future endeavors in determining the effects of manipulations such as fatigue, interruptions, and computer-aided detection.

No abstract

Interruptive clinical decision support systems, both within and outside of electronic health records, are a resource that should be used sparingly and monitored closely. Excessive use of interruptive alerting can quickly lead to alert fatigue and decreased effectiveness and ignoring of alerts. In this review, we discuss the evidence for effective alert stewardship as well as practices and methods we have found useful to assess interruptive alert burden, reduce excessive firings, optimize alert effectiveness, and establish quality governance at our institutions. We also discuss the importance of a holistic view of the alerting ecosystem beyond the electronic health record.

Background Excessive electronic health record (EHR) alerts reduce the salience of actionable alerts. Little is known about the frequency of interruptive alerts across health systems and how the choice of metric affects which users appear to have the highest alert burden. Objective (1) Analyze alert burden by alert type, care setting, provider type, and individual provider across 6 pediatric health systems. (2) Compare alert burden using different metrics. Materials and Methods We analyzed interruptive alert firings logged in EHR databases at 6 pediatric health systems from 2016–2019 using 4 metrics: (1) alerts per patient encounter, (2) alerts per inpatient-day, (3) alerts per 100 orders, and (4) alerts per unique clinician days (calendar days with at least 1 EHR log in the system). We assessed intra- and interinstitutional variation and how alert burden rankings differed based on the chosen metric. Results Alert burden varied widely across institutions, ranging from 0.06 to 0.76 firings per encounter, 0.22 to 1.06 firings per inpatient-day, 0.98 to 17.42 per 100 orders, and 0.08 to 3.34 firings per clinician day logged in the EHR. Custom alerts accounted for the greatest burden at all 6 sites. The rank order of institutions by alert burden was similar regardless of which alert burden metric was chosen. Within institutions, the alert burden metric choice substantially affected which provider types and care settings appeared to experience the highest alert burden. Conclusion Estimates of the clinical areas with highest alert burden varied substantially by institution and based on the metric used.

Objective The aim of this study was to investigate (1) why ordering clinicians use free-text orders to communicate medication information; (2) what risks physicians and nurses perceive when free-text orders are used for communicating medication information; and (3) how electronic health records (EHRs) could be improved to encourage the safe communication of medication information. Methods We performed semi-structured, scenario-based interviews with eight physicians and eight nurses. Interview responses were analyzed and grouped into common themes. Results Participants described eight reasons why clinicians use free-text medication orders, five risks relating to the use of free-text medication orders, and five recommendations for improving EHR medication-related communication. Poor usability, including reduced efficiency and limited functionality associated with structured order entry, was the primary reason clinicians used free-text orders to communicate medication information. Common risks to using free-text orders for medication communication included the increased likelihood of missing orders and the increased workload on nurses responsible for executing orders. Discussion Clinicians' use of free-text orders is primarily due to limitations in the current structured order entry design. To encourage the safe communication of medication information between clinicians, the EHR's structured order entry must be redesigned to support clinicians' cognitive and workflow needs that are currently being addressed via the use of free-text orders. Conclusion Clinicians' use of free-text orders as a workaround to insufficient structured order entry can create unintended patient safety risks. Thoughtful solutions designed to address these workarounds can improve the medication ordering process and the subsequent medication administration process.