TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods

Collins, Gary S; Moons, Karel G M; Dhiman, Paula; Riley, Richard D; Beam, Andrew L; Van Calster, Ben; Ghassemi, Marzyeh; Liu, Xiaoxuan; Reitsma, Johannes B; van Smeden, Maarten; Boulesteix, Anne-Laure; Camaradou, Jennifer Catherine; Celi, Leo Anthony; Denaxas, Spiros; Denniston, Alastair K; Glocker, Ben; Golub, Robert M; Harvey, Hugh; Heinze, Georg; Hoffman, Michael M; Kengne, André Pascal; Lam, Emily; Lee, Naomi; Loder, Elizabeth W; Maier-Hein, Lena; Mateen, Bilal A; McCradden, Melissa D; Oakden-Rayner, Lauren; Ordish, Johan; Parnell, Richard; Rose, Sherri; Singh, Karandeep; Wynants, Laure; Logullo, Patricia

doi:10.1136/bmj-2023-078378

Cited by 36 publications

(9 citation statements)

References 97 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This was a prognostic study, with the findings reported in alignment with the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis + Artificial Intelligence (TRIPOD+AI) statement [ 27 ]. Specifically, PAD-specific prognostic biomarkers were identified from a pool of circulating myokines, and these biomarkers were used in combination with relevant clinical features to develop a predictive model for adverse limb events in patients with PAD.…”

Section: Methodsmentioning

confidence: 99%

The Identification and Evaluation of Interleukin-7 as a Myokine Biomarker for Peripheral Artery Disease Prognosis

Li,

Shaikh,

Zamzam

et al. 2024

JCM

View full text Add to dashboard Cite

Background/Objectives: Myokines have been demonstrated to be associated with cardiovascular diseases; however, they have not been studied as biomarkers for peripheral artery disease (PAD). We identified interleukin-7 (IL-7) as a prognostic biomarker for PAD from a panel of myokines and developed predictive models for 2-year major adverse limb events (MALEs) using clinical features and plasma IL-7 levels. Methods: A prognostic study was conducted with a cohort of 476 patients (312 with PAD and 164 without PAD) that were recruited prospectively. Their plasma concentrations of five circulating myokines were measured at recruitment, and the patients were followed for two years. The outcome of interest was two-year MALEs (composite of major amputation, vascular intervention, or acute limb ischemia). Cox proportional hazards analysis was performed to identify IL-7 as the only myokine that was associated with 2-year MALEs. The data were randomly divided into training (70%) and test sets (30%). A random forest model was trained using clinical characteristics (demographics, comorbidities, and medications) and plasma IL-7 levels with 10-fold cross-validation. The primary model evaluation metric was the F1 score. The prognostic model was used to classify patients into low vs. high risk of developing adverse limb events based on the Youden Index. Freedom from MALEs over 2 years was compared between the risk-stratified groups using Cox proportional hazards analysis. Results: Two-year MALEs occurred in 28 (9%) of patients with PAD. IL-7 was the only myokine that was statistically significantly correlated with two-year MALE (HR 1.56 [95% CI 1.12–1.88], p = 0.007). For the prognosis of 2-year MALEs, our model achieved an F1 score of 0.829 using plasma IL-7 levels in combination with clinical features. Patients classified as high-risk by the predictive model were significantly more likely to develop MALEs over a 2-year period (HR 1.66 [95% CI 1.22–1.98], p = 0.006). Conclusions: From a panel of myokines, IL-7 was identified as a prognostic biomarker for PAD. Using a combination of clinical characteristics and plasma IL-7 levels, we propose an accurate predictive model for 2-year MALEs in patients with PAD. Our model may support PAD risk stratification, guiding clinical decisions on additional vascular evaluation, specialist referrals, and medical/surgical management, thereby improving outcomes.

show abstract

Section: Methodsmentioning

confidence: 99%

The Identification and Evaluation of Interleukin-7 as a Myokine Biomarker for Peripheral Artery Disease Prognosis

Li,

Shaikh,

Zamzam

et al. 2024

JCM

View full text Add to dashboard Cite

show abstract

“…Data analysis was performed using both Python (version 3.8.0) and R (version 4.2.2) programming languages. This study was carried out following the TRIPOD-AI guideline ( Supplemental Material 1 ) 27 . Saliency maps were generated to identify the areas of interest for the classifier across all test datasets.…”

Section: Methodsmentioning

confidence: 99%

Opportunistic Screening of Chronic Liver Disease with Deep Learning Enhanced Echocardiography

Sahashi,

Vukadinovic,

Amrollahi

et al. 2024

Preprint

View full text Add to dashboard Cite

ImportanceChronic liver disease affects more than 1.5 billion adults worldwide, however the majority of cases are asymptomatic and undiagnosed. Echocardiography is broadly performed and visualizes the liver; but this information is not leveraged.ObjectiveTo develop and evaluate a deep learning algorithm on echocardiography videos to enable opportunistic screening for chronic liver disease.DesignRetrospective observational cohortsSettingTwo large urban academic medical centersParticipantsAdult patients who received echocardiography and abdominal imaging (either abdominal ultrasound or abdominal magnetic resonance imaging) with ≤30 days between tests, between July 4, 2012, to June 4, 2022.ExposureDeep learning model predictions from a deep-learning computer vision pipeline that identifies subcostal view echocardiogram videos and detects the presence of cirrhosis or steatotic liver disease (SLD).Main Outcome and MeasuresClinical diagnosis by paired abdominal ultrasound or magnetic resonance imaging (MRI).ResultsA total of 1,596,640 echocardiogram videos (66,922 studies from 24,276 patients) from Cedars-Sinai Medical Center (CSMC) were used to develop EchoNet-Liver, an automated pipeline that identifies high quality subcostal images from echocardiogram studies and detects the presence of cirrhosis or SLD. In the held-out CSMC test cohort, EchoNet-Liver was able to detect the presence of cirrhosis with an AUC of 0.837 (0.789 - 0.880) and SLD with an AUC of 0.799 (0.758 - 0.837). In a separate test cohort with paired abdominal MRIs, cirrhosis was detected with an AUC of 0.704 (0.689-0.718) and SLD was detected with an AUC of 0.726 (0.659-0.790). In an external test cohort of 106 patients (n = 5,280 videos), the model detected cirrhosis with an AUC of 0.830 (0.738 - 0.909) and SLD with an AUC of 0.768 (0.652 – 0.875).Conclusions and RelevanceDeep learning assessment of clinical echocardiography enables opportunistic screening of SLD and cirrhosis. Application of this algorithm may identify patients who may benefit from further diagnostic testing and treatment for chronic liver disease.KEY POINTSQuestionCan a deep learning algorithm applied to echocardiography videos effectively identify chronic liver diseases including cirrhosis and steatotic liver disease (SLD)?FindingsThis retrospective observational cohort study utilized 1,596,640 echocardiography videos from 66,922 studies of 24,276 patients. The deep learning model with a computer vision pipeline (EchoNet-Liver) demonstrated strong performance to detect cirrhosis and SLD. External validation at a geographically distinct site demonstrated similar discriminative ability.MeaningThe application of EchoNet-Liver to echocardiography could aid opportunistic screening of chronic liver diseases, providing a unique cost-effective angle to improve patient management.

show abstract

“…Reporting Quality Assessment: To assess the quality of the reporting, we decided to use TRIPOD+AI statement, which contains a comprehensive list of items that need to be reported for papers reporting development and/or validation of prognostic AI model [43]. List of sections and items on this list covers every key part of a manuscript including title, abstract, introduction, methods, results, and discussion.…”

Section: Methodologic Quality Assessmentmentioning

confidence: 99%

A Systematic Review of Machine Learning-based Prognostic Models for Acute Pancreatitis: Towards Improving Methods and Reporting Quality

Critelli,

Hassan,

Lahooti

et al. 2024

Preprint

Self Cite

View full text Add to dashboard Cite

BackgroundAn accurate prognostic tool is essential to aid clinical decision making (e.g., patient triage) and to advance personalized medicine. However, such prognostic tool is lacking for acute pancreatitis (AP). Increasingly machine learning (ML) techniques are being used to develop high-performing prognostic models in AP. However,methodologic and reporting quality has received little attention. High-quality reporting and study methodology are critical to model validity, reproducibility, and clinical implementation. In collaboration with content experts in ML methodology, we performed a systematic review critically appraising the quality of methodology and reporting of recently published ML AP prognostic models.MethodsUsing a validated search strategy, we identified ML AP studies from the databases MEDLINE, PubMed, and EMBASE published between January 2021 and December 2023. Eligibility criteria included all retrospective or prospective studies that developed or validated new or existing ML models in patients with AP that predicted an outcome following an episode of AP. Meta-analysis was considered if there was homogeneity in the study design and in the type of outcome predicted. For risk of bias (ROB) assessment, we used the Prediction Model Risk of Bias Assessment Tool (PROBAST). Quality of reporting was assessed using the Transparent Reporting of a Multivariable Prediction Model of Individual Prognosis or Diagnosis – Artificial Intelligence (TRIPOD+AI) statement that defines standards for 27 items that should be reported in publications using ML prognostic models.ResultsThe search strategy identified 6480 publications of which 30 met the eligibility criteria. Studies originated from China (22), U.S (4), and other (4). All 30 studies developed a new ML model and none sought to validate an existing ML model, producing a total of 39 new ML models. AP severity (23/39) or mortality (6/39) were the most common outcomes predicted. The mean area-under-the-curve for all models and endpoints was 0.91 (SD 0.08). The ROB was high for at least one domain in all 39 models, particularly for the analysis domain (37/39 models). Steps were not taken to minimize over-optimistic model performance in 27/39 models. Due to heterogeneity in the study design and in how the outcomes were defined and determined, meta-analysis was not performed.Studies reported on only 15/27 items from TRIPOD+AI standards, with only 7/30 justifying sample size and 13/30 assessing data quality. Other reporting deficiencies included omissions regarding human-AI interaction (28/30), handling low-quality or incomplete data in practice (27/30), sharing analytical codes (25/30), study protocols (25/30) and reporting source data (19/30),.DiscussionThere are significant deficiencies in the methodology and reporting of recently published ML based prognostic models in AP patients. These undermine the validity, reproducibility and implementation of these prognostic models despite their promise of superior predictive accuracy.FundingnoneRegistrationResearch Registry (reviewregistry1727)

show abstract

TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods

Cited by 36 publications

References 97 publications

The Identification and Evaluation of Interleukin-7 as a Myokine Biomarker for Peripheral Artery Disease Prognosis

The Identification and Evaluation of Interleukin-7 as a Myokine Biomarker for Peripheral Artery Disease Prognosis

Opportunistic Screening of Chronic Liver Disease with Deep Learning Enhanced Echocardiography

A Systematic Review of Machine Learning-based Prognostic Models for Acute Pancreatitis: Towards Improving Methods and Reporting Quality

Contact Info

Product

Resources

About