Objectives: This paper draws attention to: i) key considerations for evaluating artificial intelligence (AI) enabled clinical decision support; and ii) challenges and practical implications of AI design, development, selection, use, and ongoing surveillance.
Method: A narrative review of existing research and evaluation approaches along with expert perspectives drawn from the International Medical Informatics Association (IMIA) Working Group on Technology Assessment and Quality Development in Health Informatics and the European Federation for Medical Informatics (EFMI) Working Group for Assessment of Health Information Systems.
Results: There is a rich history and tradition of evaluating AI in healthcare. While evaluators can learn from past efforts, and build on best practice evaluation frameworks and methodologies, questions remain about how to evaluate the safety and effectiveness of AI that dynamically harness vast amounts of genomic, biomarker, phenotype, electronic record, and care delivery data from across health systems. This paper first provides a historical perspective about the evaluation of AI in healthcare. It then examines key challenges of evaluating AI-enabled clinical decision support during design, development, selection, use, and ongoing surveillance. Practical aspects of evaluating AI in healthcare, including approaches to evaluation and indicators to monitor AI are also discussed.
Conclusion: Commitment to rigorous initial and ongoing evaluation will be critical to ensuring the safe and effective integration of AI in complex sociotechnical settings. Specific enhancements that are required for the new generation of AI-enabled clinical decision support will emerge through practical application.
BackgroundThe relative contribution of long term care facilities (LTCFs) and hospitals in the transmission of methicillin-resistant Staphylococcus aureus (MRSA) is unknown.MethodsConcurrent MRSA screening and spa type analysis was performed in LTCFs and their network hospitals to estimate the rate of MRSA acquisition among residents during their stay in LTCFs and hospitals, by colonization pressure and MRSA transmission calculations.ResultsIn 40 LTCFs, 436 (21.6%) of 2020 residents were identified as ‘MRSA-positive’. The incidence of MRSA transmission per 1000-colonization-days among the residents during their stay in LTCFs and hospitals were 309 and 113 respectively, while the colonization pressure in LTCFs and hospitals were 210 and 185 per 1000-patient-days respectively. MRSA spa type t1081 was the most commonly isolated linage in both LTCF residents (76/121, 62.8%) and hospitalized patients (51/87, 58.6%), while type t4677 was significantly associated with LTCF residents (24/121, 19.8%) compared with hospitalized patients (3/87, 3.4%) (p < 0.001). This suggested continuous transmission of MRSA t4677 among LTCF residents. Also, an inverse linear relationship between MRSA prevalence in LTCFs and the average living area per LTCF resident was observed (Pearson correlation −0.443, p = 0.004), with the odds of patients acquiring MRSA reduced by a factor of 0.90 for each 10 square feet increase in living area.ConclusionsOur data suggest that MRSA transmission was more serious in LTCFs than in hospitals. Infection control should be focused on LTCFs in order to reduce the burden of MRSA carriers in healthcare settings.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.