A framework for the oversight and local deployment of safe and high-quality prediction models

Abstract

Artificial intelligence/machine learning models are being rapidly developed and used in clinical practice. However, many models are deployed without a clear understanding of clinical or operational impact and frequently lack monitoring plans that can detect potential safety signals. There is a lack of consensus in establishing governance to deploy, pilot, and monitor algorithms within operational healthcare delivery workflows. Here, we describe a governance framework that combines current regulatory best practices and lifecycle management of predictive models being used for clinical care. Since January 2021, we have successfully added models to our governance portfolio and are currently managing 52 models.

Guiding Principles to Address the Impact of Algorithm Bias on Racial and Ethnic Disparities in Health and Health Care

“…Transparency in the types of training data, processes, and evaluations used is paramount. For example, an academic medical center recently published its framework for oversight and deployment of prediction models, which includes checkpoint gates and an oversight governance structure . Current evidence suggests that such governance infrastructure is rare …”

President Biden’s Executive Order on Artificial Intelligence—Implications for Health Care Organizations

“…Stanford Health Care (SHC) and Duke Health, for instance, both require all AI tools proposed for clinical use in their facilities to undergo review by an oversight group specifically constituted for that purpose. 6,7 In SHC's FURM (Fair, Useful, Reliable Model) review process, a team of data scientists, ethicists, clinicians, and administrators conducts a broad evaluation of proposed AI uses. 7 For uses greenlighted for deployment, follow-up assessments monitor how the concerns raised are addressed.…”

“…7 For uses greenlighted for deployment, follow-up assessments monitor how the concerns raised are addressed. The ethical review process includes quantitative assessments of model fairness for specific patient subgroups, qualitativeinterviewingwithmultiplestakeholders(AIdevelopers,clinical staff, hospital administrators, patient representatives) to surface ethical concerns, 8 and consultation with additional AI experts about recommendationsforaddressingtheseconcerns.DukeHealth'sABCDS (Algorithm-Based Clinical Decision Support) process requires proposers to develop and submit plans demonstrating the algorithm's clinical utility that include assessments of model fairness, 6 and work is underway to operationalize a wider range of ethical assessment criteria.…”

“…However, some academic centers have begun to develop centralized processes of AI review that include ethics assessments. Stanford Health Care (SHC) and Duke Health, for instance, both require all AI tools proposed for clinical use in their facilities to undergo review by an oversight group specifically constituted for that purpose . In SHC’s FURM (Fair, Useful, Reliable Model) review process, a team of data scientists, ethicists, clinicians, and administrators conducts a broad evaluation of proposed AI uses .…”

The Need for Targeted Labeling of Machine Learning–Based Software as a Medical Device

“…In response to this, institutions, such as Duke University, are establishing governance frameworks for deploying CDS tools. 5 A key aspect of our process is the local testing of CDS tools and assessing their performance across and within diverse patient populations before deployment. We view this as an important step to ensuring the high quality and equitable performance of any CDS.…”