Evaluating visual analytics for health informatics applications: a systematic review from the American Medical Informatics Association Visual Analytics Working Group Task Force on Evaluation

Abstract

Objective This article reports results from a systematic literature review related to the evaluation of data visualizations and visual analytics technologies within the health informatics domain. The review aims to (1) characterize the variety of evaluation methods used within the health informatics community and (2) identify best practices. Methods A systematic literature review was conducted following PRISMA guidelines. PubMed searches were conducted in February 2017 using search terms representing key concepts of interest: health care settings, visualization, and evaluation. References were also screened for eligibility. Data were extracted from included studies and analyzed using a PICOS framework: Participants, Interventions, Comparators, Outcomes, and Study Design. Results After screening, 76 publications met the review criteria. Publications varied across all PICOS dimensions. The most common audience was healthcare providers (n = 43), and the most common data gathering methods were direct observation (n = 30) and surveys (n = 27). About half of the publications focused on static, concentrated views of data with visuals (n = 36). Evaluations were heterogeneous regarding setting and measurements used. Discussion When evaluating data visualizations and visual analytics technologies, a variety of approaches have been used. Usability measures were used most often in early (prototype) implementations, whereas clinical outcomes were most common in evaluations of operationally-deployed systems. These findings suggest opportunities for both (1) expanding evaluation practices, and (2) innovation with respect to evaluation methods for data visualizations and visual analytics technologies across health settings. Conclusion Evaluation approaches are varied. New studies should adopt commonly reported metrics, context-appropriate study designs, and phased evaluation strategies.

Advancing biomedical and health informatics knowledge through reviews of existing research

“…For example, Koleck et al 8 focused on use of natural language processing to process or analyze information related to a selected set of symptoms in electronic health record free-text narratives while other systematic reviews examined broader topics such as clinical pathways, 9 speech recognition technology for clinician documentation, 10 and evaluation approaches for visual analytic technologies in health. 6…”

“…The manuscript should also provide details about how the process was managed. The methods varied in our 5 examples, ranging from author-developed spreadsheets 6 and specialized software such as Covidence from the Cochrane Collaboration (https://community.cochrane.org/help/tools-and-software/covidence). 8…”

“…Similarly, Koleck et al 8 reported on the number of documents and a set of metrics suitable for evaluating an natural language processing algorithm or pipeline including sensitivity, specificity, precision, recall, F measure, kappa statistic, area under the receiver-operating characteristic curve, and C-statistic. In 1 instance, 6 the actual purposes of the review were to discover what evaluation measures were used in the literature and to make recommendations related to quality assessments. I encourage authors to visit the EQUATOR (Enhancing the QUAlity and Transparency Of health Research) network website (https://www.equator-network.org/reportingguidelines/) to retrieve reporting guidelines for a variety of study designs to assist with selection of quality assessment criteria.…”

“…However, 3 reviews contained graphical components, including a histogram of studies over time, 10 a chord diagram showing the associations between symptoms and studies, 8 and a heatmap displaying the relationships among PICOS (participants, interventions, comparisons, outcomes, and study design) 7 components and setting and measurement classifications. 6 Now that JAMIA is an online journal, there are no charges for color, so I encourage authors to include graphs when appropriate.…”

“…5 This issue of JAMIA has 5 systematic reviews including 1 from an American Medical Informatics Association Working Group. 6 In this editorial, I highlight how these systematic reviews demonstrate their relevance to the JAMIA audience and meet best-practice standards for systematic reviews as delineated in the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement 7 and other sources.…”