Prediction of High-Risk Donors for Kidney Discard and Nonrecovery Using Structured Donor Characteristics and Unstructured Donor Narratives

Abstract

ImportanceDespite the unmet need, many deceased-donor kidneys are discarded or not recovered. Inefficient allocation and prolonged ischemia time are contributing factors, and early detection of high-risk donors may reduce organ loss.ObjectiveTo evaluate the feasibility of machine learning (ML) and natural language processing (NLP) classification of donors with kidneys that are used vs not used for organ transplant.Design, Setting, and ParticipantsThis retrospective cohort study used donor information (structured donor characteristics and unstructured donor narratives) from the United Network for Organ Sharing (UNOS). All donor offers to a single transplant center between January 2015 and December 2020 were used to train and validate ML models to predict donors who had at least 1 kidney transplanted (at our center or another center). The donor data from 2021 were used to test each model.ExposuresDonor information was provided by UNOS to the transplant centers with potential transplant candidates. Each center evaluated the donor and decided within an allotted time whether to accept the kidney for organ transplant.Main Outcomes and MeasuresOutcome metrics of the test cohort included area under the receiver operating characteristic curve (AUROC), F1 score, accuracy, precision, and recall of each ML classifier. Feature importance and Shapley additive explanation (SHAP) summaries were assessed for model explainability.ResultsThe training/validation cohort included 9555 donors (median [IQR] age, 50 [36-58] years; 5571 male [58.3%]), and the test cohort included 2481 donors (median [IQR] age, 52 [40-59] years; 1496 male [60.3%]). Only 20% to 30% of potential donors had at least 1 kidney transplanted. The ML model with a single variable (Kidney Donor Profile Index) showed an AUROC of 0.69, F1 score of 0.42, and accuracy of 0.64. Multivariable ML models based on basic a priori structured donor data showed similar metrics (logistic regression: AUROC = 0.70; F1 score = 0.42; accuracy = 0.62; random forest classifier: AUROC = 0.69; F1 score = 0.42; accuracy = 0.64). The classic NLP model (bag-of-words model) showed its best metrics (AUROC = 0.60; F1 score = 0.35; accuracy = 0.59) by the logistic regression classifier. The advanced Bidirectional Encoder Representations From Transformers model showed comparable metrics (AUROC = 0.62; F1 score = 0.39; accuracy = 0.69) only after appending basic donor information. Feature importance and SHAP detected the variables (and words) that affected the models most.Conclusions and RelevanceResults of this cohort study suggest that models using ML can be applied to predict donors with high-risk kidneys not used for organ transplant, but the models still need further elaboration. The use of unstructured data is likely to expand the possibilities; further exploration of new approaches will be necessary to develop models with better predictive metrics.

Concerns Over Prediction of Kidney Discard and Nonrecovery

“…It is widely recognized that proper validation and testing of trained models is necessary to avoid overfitting and poor generalizability . Surprisingly, trained ML models were decently testable in this study, which makes the possibility of chance findings an open question and merits special consideration. To account for the imbalance in the 2 datasets, propensity score matching is recommended.…”

“…Additionally, there is evidence that SARS-CoV-2 infection has a significant effect on kidney transplant activity . Because the testing dataset involved donors who tested positive for COVID-19 in this study, it remains questionable for the validity and explainability of trained models.…”

“…To the Editor Sageshima et al have proven the feasibility of modeling structured and unstructured data via machine learning (ML) and natural language processing (NLP) to predict donors with high risk for kidney discard and nonrecovery. Although model performance is suboptimal, the study by Sageshima et al represents a step forward toward the implementation of artificial intelligence into clinical practice and underscores the necessity of incorporating structured and unstructured data to boost model robustness. We applaud the authors’ outstanding work; however, we have 2 concerns over data presentation and interpretation.…”

The Explainability of Transformers: Current Status and Directions

“…Rizinski et al proposed a framework to perform a lexicon-based sentiment analysis [134]. Sageshima et al proposed a method to classify donors with high-risk kidneys in [135].…”

Application of artificial intelligence to hepatobiliary cancer clinical outcomes research

“…For instance, AI algorithms such as Virtual Twin analysis or Optimal Policy Tree (OPT) [11,12] can differentiate patients into subsets, identifying who may benefit from specific treatment plans, such as upfront surgery versus neoadjuvant or adjuvant chemotherapy [13,14] . Moreover, AI can facilitate drug discovery and selection of chemotherapeutic agents by analyzing massive datasets to predict the most promising drug candidates, as well as potentially optimizing clinical trial designs, thereby reducing the time and cost of developing new drugs to market [15] . Natural Language Processing (NLP) algorithms can also sift through unstructured healthcare data, distilling valuable insights in the electronic health records and medical literature, thus enhancing the accessibility of critical information for healthcare providers and researchers [16] .…”