Implementation of Machine Learning Pipelines for Clinical Practice: Development and Validation Study

Abstract

Background Artificial intelligence (AI) technologies, such as machine learning and natural language processing, have the potential to provide new insights into complex health data. Although powerful, these algorithms rarely move from experimental studies to direct clinical care implementation. Objective We aimed to describe the key components for successful development and integration of two AI technology–based research pipelines for clinical practice. Methods We summarized the approach, results, and key learnings from the implementation of the following two systems implemented at a large, tertiary care children’s hospital: (1) epilepsy surgical candidate identification (or epilepsy ID) in an ambulatory neurology clinic; and (2) an automated clinical trial eligibility screener (ACTES) for the real-time identification of patients for research studies in a pediatric emergency department. Results The epilepsy ID system performed as well as board-certified neurologists in identifying surgical candidates (with a sensitivity of 71% and positive predictive value of 77%). The ACTES system decreased coordinator screening time by 12.9%. The success of each project was largely dependent upon the collaboration between machine learning experts, research and operational information technology professionals, longitudinal support from clinical providers, and institutional leadership. Conclusions These projects showcase novel interactions between machine learning recommendations and providers during clinical care. Our deployment provides seamless, real-time integration of AI technology to provide decision support and improve patient care.

Continuous Data-Driven Monitoring in Critical Congenital Heart Disease: Clinical Deterioration Model Development

“…Several steps must be taken to progress this model—and others alike—toward implementation in daily clinical practice [ 28 , 29 ]. Primarily, a data infrastructure is required to enable real-time or near–real-time data availability to AI models, allowing their prospective validation.…”

A comprehensive review of machine learning algorithms and their application in geriatric medicine: present and future

“…Improved performance from ML is often achieved at the expense of increased model complexity, resulting in uncertainty regarding both the way they operate mathematically and in their predictions and decisions [ 5 ]. This so-called “black box” nature of complex algorithms which is often the trade-off for enhanced performance is often identified as the key for a lack of trust by clinicians and creates barriers for the adoption and implementation of AI into healthcare [ 176 , 177 ]. DL techniques in particular are especially known for having limited transparency [ 178 ], which is a direct result of the non-linear structure and transformation of the original data during error back-propagation, an inherent feature of DL algorithms [ 19 ].…”

Automated, machine learning–based alerts increase epilepsy surgery referrals: A randomized controlled trial

“…22 Only limited studies illustrate the direct application of NLP to clinical practice or clinical decision support systems. [22][23][24][25][26] In the case of drug-resistent epilepsy, NLP can identify eligible patients and suggest neurosurgical consults earlier in the disease process. 27,28 Our group developed an NLP model to analyze freetext neurology office visit notes to identify potential candidates for resective epilepsy surgery.…”

“…NLP can be applied to evaluate clinical notes and provide recommendations, 21 but NLP models are frequently experimental and not integrated into practice 22 . Only limited studies illustrate the direct application of NLP to clinical practice or clinical decision support systems 22–26 . In the case of drug‐resistent epilepsy, NLP can identify eligible patients and suggest neurosurgical consults earlier in the disease process 27,28 …”