Using artificial intelligence in medical school admissions screening to decrease inter- and intra-observer variability

Abstract

Objectives Inter- and intra-observer variability is a concern for medical school admissions. Artificial intelligence (AI) may present an opportunity to apply a fair standard to all applicants systematically and yet maintain sensitivity to nuances that have been a part of traditional screening methods. Material and Methods Data from 5 years of medical school applications were retrospectively accrued and analyzed. The applicants (m = 22 258 applicants) were split 60%–20%–20% into a training set (m = 13 354), validation set (m = 4452), and test set (m = 4452). An AI model was trained and evaluated with the ground truth being whether a given applicant was invited for an interview. In addition, a “real-world” evaluation was conducted simultaneously within an admissions cycle to observe how it would perform if utilized. Results The algorithm had an accuracy of 95% on the training set, 88% on the validation set, and 88% on the test set. The area under the curve of the test set was 0.93. The SHapely Additive exPlanations (SHAP) values demonstrated that the model utilizes features in a concordant manner with current admissions rubrics. By using a combined human and AI evaluation process, the accuracy of the process was demonstrated to be 96% on the “real-world” evaluation with a negative predictive value of 0.97. Discussion and Conclusion These results demonstrate the feasibility of an AI approach applied to medical school admissions screening decision-making. Model explainability and supplemental analyses help ensure that the model makes decisions as intended.

Exploring Depression and Nutritional Covariates Amongst US Adults using Shapely Additive Explanations

“…These transparent machine‐learning tools allow for increased confidence that these algorithms are picking up true signal within these covariates to predict the presence of depression rather than just replicating potential biases stemming from systemic data‐quality errors that are present within the data set. Additionally, these SHAP visualizations allow us to interpret that the increase predictive power of these machine‐learning methods is associated with the ability for these nonparametric methods to more accurately capture the nonlinear interactive relationship between the covariates, rather than just over‐fitting the model to get increased accuracy 52,53 …”

Use of feature importance statistics to accurately predict asthma attacks using machine learning: A cross-sectional cohort study of the US population

“…By using transparent machine learning tools, we can ensure that the model is detecting genuine signals within these covariates to predict asthma attacks, rather than simply replicating biases present in the dataset [32][33][34][35]. The SHAP visualizations further support the increased predictive power of these non-parametric methods by demonstrating their ability to accurately capture the non-linear interactions between covariates, without overfitting the model to achieve greater accuracy [20,[36][37][38][39].…”

Utilization of ChatGPT for Appraising Letters of Recommendation in Urology Residency Applications: Ready for Prime Time?

“…Some institutions have developed algorithms incorporating diverse applicant attributes to provide more holistic reviews. 6,7 Although it may be tempting to utilize these algorithms, there is concern that new biases may be introduced if not carefully customized.…”