De-black-boxing health AI: demonstrating reproducible machine learning computable phenotypes using the N3C-RECOVER Long COVID model in the All of Us data repository

Abstract

Machine learning (ML)-driven computable phenotypes are among the most challenging to share and reproduce. Despite this difficulty, the urgent public health considerations around Long COVID make it especially important to ensure the rigor and reproducibility of Long COVID phenotyping algorithms such that they can be made available to a broad audience of researchers. As part of the NIH Researching COVID to Enhance Recovery (RECOVER) Initiative, researchers with the National COVID Cohort Collaborative (N3C) devised and trained an ML-based phenotype to identify patients highly probable to have Long COVID. Supported by RECOVER, N3C and NIH’s All of Us study partnered to reproduce the output of N3C’s trained model in the All of Us data enclave, demonstrating model extensibility in multiple environments. This case study in ML-based phenotype reuse illustrates how open-source software best practices and cross-site collaboration can de-black-box phenotyping algorithms, prevent unnecessary rework, and promote open science in informatics.

Reengineering a machine learning phenotype to adapt to the changing COVID-19 landscape: A study from the N3C and RECOVER consortia

“…Non-symptom-related, continuous variables such as healthcare utilization rates are binned to avoid overfitting and overreliance on patterns unique to the N3C population. As was true of the prior version, 12 we believe this new model version will be translatable to other sites and consortia with OMOP implementations to promote reuse and reproducibility.…”

“…11 The ML model's purpose is to use information from EHR data to predict missing Long COVID labels, thus serving as a computable phenotype for Long COVID. The model was performant and generalizable, 12 but heavily relies on the existence and timing of an index date for a patient's acute COVID-19 infection. Moreover, the model only considers a patient's first SARS-CoV-2 infection, as at the time we did not anticipate the now-common occurrence of SARS-CoV-2 reinfections.…”

Genetic and Survey Data Improves Performance of Machine Learning Model for Long COVID

“…The group developed an advanced machine learning (ML) based phenotype for long COVID and tested it using the NIH’s All of Us (AoU) data. 3 , 4 While an important step in identifying long COVID patients, the model relies entirely on patients’ EHR data, which is only representative of a patient’s recorded history in a medical setting. This potentially limits the algorithm’s ability to detect cases.…”