Predicting Medical Student Success on Licensure Exams

PhD, Charles A. Gullo; McCarthy, Michael; Shapiro, Joseph I.; Miller, Bobby

doi:10.1007/s40670-015-0179-6

Cited by 16 publications

(15 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Even combined, their correlation with STEP1 is weak at best and nonexistent at worst. Similar observations were made in our earlier studies (Gullo et al 2015). However, prospective application of prediction modeling in this study shows that traditionally used pre-matriculation data is all but useless in predicting STEP1 scores for the candidates.…”

Section: Discussion: -supporting

confidence: 89%

“…The nal key nding of this study is the validity and reliability of our STEP1 prediction model generated using targeted assessments, including the course-end NBMEs. As opposed to the retrospective model published by us (Gullo et al 2015), and others(Coumarbatch et al 2010) , (Giordano et al 2016), this model is effective prospectively. The model is deployed to identify at-risk students early and provide intervention if needed.…”

Section: Discussion: -mentioning

confidence: 82%

“…However, one challenge JCESOM has encountered is accurately predicting a medical student's performance on USMLE STEP 1, despite utilizing prediction modeling (Gullo et al 2015). For example, students who excel in the rst two years of PreClerkship education and are predicted to pass USMLE STEP 1 fail or underperform, and vice versa.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Validity and Reliability of Institutional Assessments in Predicting USMLE STEP 1 Scores: Lessons from a Traditional 2 x 2 Curricular Model

Puri

Graham-Smith

McCarthy

et al. 2020

Preprint

View full text Add to dashboard Cite

Purpose: We have observed that students’ performance in our PreClerkship curriculum does not align well with their USMLE STEP1 scores. Students at-risk of failing or underperforming on STEP1 have often excelled in our institutional assessments. We sought to test the validity and reliability of our course-assessments in predicting STEP1 scores, and in the process generate and validate a more accurate prediction model for STEP1 performance.Methods: Student pre-matriculation and course assessment data of the Class of 2020 (n = 76) is used to generate a stepwise STEP1 prediction model, which is tested with the students of the Class of 2021 (n = 71). Predictions are generated for the end of each course in the programing language R. For the Class of 2021, predicted STEP1 score is correlated with their actual STEP1 scores and data-agreement is tested with means-difference plots. A similar model is generated and tested for the Class of 2022.Results: STEP1 predictions based pre-matriculation data are unreliable and fail to identify at-risk students (R2 = 0.02). STEP1 predictions for most year 1 courses (anatomy, biochemistry, physiology) correlate poorly with students’ actual STEP1 scores (R2 = 0.30). STEP1 predictions improve for year 2 courses (microbiology, pathology and pharmacology), but reliable predictions are based on truly integrated courses with customized NBMEs as comprehensive exams (0.66). Predictions based on these NBMEs and integrated courses are reproducible for the Class of 2022.Conclusion: MCAT and undergraduate GPA are poor predictors of students’ STEP1 scores. Partially integrated courses with biweekly assessments do not promote problem-solving skills and leave students’ at-risk of failing STEP1. Only truly integrated courses with comprehensive assessments are reliable indicators of students STEP1 preparation.

show abstract

Section: Discussion: -supporting

confidence: 89%

Section: Discussion: -mentioning

confidence: 82%

See 1 more Smart Citation

Validity and Reliability of Institutional Assessments in Predicting USMLE STEP 1 Scores: Lessons from a Traditional 2 x 2 Curricular Model

Puri

Graham-Smith

McCarthy

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…In addition, we examined the utility of a support vector machine (SVM) which involves the multidimensional sorting of data based on the development of a hyperplane which best segregates the two classes. 8 Using the CARET package, 9 we employed two tuning parameters to control the performance of the SVM: kernel and C. Kernel is a complex function, which takes input from a lower dimension and transforms it to a higher dimension, and is useful in a nonlinear separation problem. We used the radial kernel option from the CARET package.…”

Section: Logistic Regression and Support Vector Machinementioning

confidence: 99%

<p>Prediction of cardiovascular outcomes with machine learning techniques: application to the Cardiovascular Outcomes in Renal Atherosclerotic Lesions (CORAL) study</p>

Chen¹,

Brewster²,

Dworkin³

et al. 2019

IJNRD

View full text Add to dashboard Cite

Background: Data derived from the Cardiovascular Outcomes in Renal Atherosclerotic Lesions (CORAL) study were analyzed in an effort to employ machine learning methods to predict the composite endpoint described in the original study. Methods: We identified 573 CORAL subjects with complete baseline data and the presence or absence of a composite endpoint for the study. These data were subjected to several models including a generalized linear (logistic-linear) model, support vector machine, decision tree, feed-forward neural network, and random forest, in an effort to attempt to predict the composite endpoint. The subjects were arbitrarily divided into training and testing subsets according to an 80%:20% distribution with various seeds. Prediction models were optimized within the CARET package of R. Results: The best performance of the different machine learning techniques was that of the random forest method which yielded a receiver operator curve (ROC) area of 68.1%±4.2% (mean ± SD) on the testing subset with ten different seed values used to separate training and testing subsets. The four most important variables in the random forest method were SBP, serum creatinine, glycosylated hemoglobin, and DBP. Each of these variables was also important in at least some of the other methods. The treatment assignment group was not consistently an important determinant in any of the models. Conclusion: Prediction of a composite cardiovascular outcome was difficult in the CORAL population, even when employing machine learning methods. Assignment to either the stenting or best medical therapy group did not serve as an important predictor of composite outcome.

show abstract

“…Even the first exam in medical school alone was a better predictor than the preadmission variables. 6 Given that the MCAT is a poor predictor of MLE performance and MLE performance yields, and at best only a partial assessment of a physician's abilities, you would think the MCAT would be devalued. Although some medical schools have dropped the requirement 7 , there is blind faith on the part of most schools that the new format of the MCAT will provide information lacking in previous versions.…”

Section: On Watson Racism and Standardized Testsmentioning

confidence: 99%

On Watson, Racism, and Standardized Tests

Shapiro¹

2019

Marshall Journal of Medicine

View full text Add to dashboard Cite

Predicting Medical Student Success on Licensure Exams

Cited by 16 publications

References 17 publications

Validity and Reliability of Institutional Assessments in Predicting USMLE STEP 1 Scores: Lessons from a Traditional 2 x 2 Curricular Model

Validity and Reliability of Institutional Assessments in Predicting USMLE STEP 1 Scores: Lessons from a Traditional 2 x 2 Curricular Model

<p>Prediction of cardiovascular outcomes with machine learning techniques: application to the Cardiovascular Outcomes in Renal Atherosclerotic Lesions (CORAL) study</p>

On Watson, Racism, and Standardized Tests

Contact Info

Product

Resources

About