Performance on the Nephrology In-Training Examination and ABIM Nephrology Certification Examination Outcomes

Jurich, Daniel; Duhigg, Lauren M; Plumb, Troy J.; Haist, Steven A.; Hawley, Janine L.; Lipner, Rebecca S.; Smith, Laurel; Norby, Suzanne M.

doi:10.2215/cjn.05580517

Cited by 15 publications

(12 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This suggests that, despite correlations between ITE and board performance, prediction of board examination pass/fail using the ITE for an individual resident is of little practical benefit. 26 Even residents who perform very poorly on the ITE have a reasonable likelihood of passing their board examination.…”

Section: Discussionmentioning

confidence: 99%

“…[16][17][18][19][20][21][22][23][24] Eleven of the 12 studies using linear regression found that ITE scores significantly predicted board examination performance. 4,7,9,10,13,[25][26][27][28][29] Only 1 study showing signicant prediction for PGY-3-PGY-4 residents, but not PGY-1-PGY-2 residents (Swanson et al, orthopaedic surgery 21 ). For logistic regression analysis, studies either used ITE scores as a predictor on a continuous scale or categorized ITE scores into 2 categories (eg, , 10th percentile, .…”

Section: Relationship To Other Variables' Validity Evidencementioning

confidence: 99%

“…For predicting a board examination passing outcome, 6 studies showed ITE scores significantly predicted who would pass the board examination. 4,9,13,26,27,29 Three additional studies showed that a particular high score, quartile, or stanine significantly predicted who would pass the board examination (Pucas 2012, otolaryngology 30 ), along with AUC good accuracy/predictive value (Lingenfelter et al, obstetrics and gynecology 31 ; Pucas 2018, otolaryngology 32 ). O'Neill et al (family medicine) 14 also found good AUC accuracy/predictive value for a particular high ITE score.…”

Section: Relationship To Other Variables' Validity Evidencementioning

confidence: 99%

See 2 more Smart Citations

A Systematic Review of the Relationship Between In-Training Examination Scores and Specialty Board Examination Scores

McCrary

Colbert-Getz

Poss

et al. 2020

Journal of Graduate Medical Education

View full text Add to dashboard Cite

Background In-training examinations (ITEs) are intended for low-stakes, formative assessment of residents' knowledge, but are increasingly used for high-stake purposes, such as to predict board examination failures. Objective The aim of this review was to investigate the relationship between performance on ITEs and board examination performance across medical specialties. Methods A search of the literature for studies assessing the strength of the relationship between ITE and board examination performance from January 2000 to March 2019 was completed. Results were categorized based on the type of statistical analysis used to determine the relationship between ITE performance and board examination performance. Results Of 1407 articles initially identified, 89 articles underwent full-text review, and 32 articles were included in this review. There was a moderate-strong relationship between ITE and board examination performance, and ITE scores significantly predict board examination scores for the majority of studies. Performing well on an ITE predicts a passing outcome for the board examination, but there is less evidence that performing poorly on an ITE will result in failing the associated specialty board examination. Conclusions There is a moderate to strong correlation between ITE performance and subsequent performance on board examinations. That the predictive value for passing the board examination is stronger than the predictive value for failing calls into question the “common wisdom” that ITE scores can be used to identify “at risk” residents. The graduate medical education community should continue to exercise caution and restraint in using ITE scores for moderate to high-stakes decisions.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Relationship To Other Variables' Validity Evidencementioning

confidence: 99%

Section: Relationship To Other Variables' Validity Evidencementioning

confidence: 99%

See 1 more Smart Citation

A Systematic Review of the Relationship Between In-Training Examination Scores and Specialty Board Examination Scores

McCrary

Colbert-Getz

Poss

et al. 2020

Journal of Graduate Medical Education

View full text Add to dashboard Cite

show abstract

“…(2) to determine interrater scoring agreement between programs and WRNMMC investigators; (3) to determine overall and scenario pass percentages and mean scores, hypothesizing that second-year fellows (third-years analyzed as second-years) would perform better than first-year fellows, and fellows in programs testing in 2016 would improve in 2017, and perform better than those at programs administering the test for the first time; (4) to identify evidence-based questions incorrectly answered by .50% of second-year fellows; (5) to determine fellow satisfaction with the OSCE as a formative evaluation tool; (6) to determine whether OSCE score correlated with in-training examination score for second-year fellows (3,18).…”

Section: Fellow Testingmentioning

confidence: 99%

The Acute Dialysis Orders Objective Structured Clinical Examination (OSCE)

Prince¹,

Nee²,

Yuan³

2019

CJASN

View full text Add to dashboard Cite

Background and objectivesAcute kidney replacement therapy (KRT) prescription is a critical nephrology skill. We administered a formative objective structured clinical examination (OSCE) to nephrology fellows to assess acute KRT medical knowledge, patient care, and systems-based practice competencies.Design, setting, participants, & measurementsProspective cohort study of an educational test using the unified model of construct validity. We tested 117 fellows: 25 (four programs) in 2016 and 92 (15 programs) in 2017; 51 first-year and 66 second-year fellows. Using institutional protocols and order sets, fellows wrote orders and answered open-ended questions on a three-scenario OSCE, previously validated by board-certified, practicing clinical nephrologists. Outcomes were overall and scenario pass percentage and score; percent correctly answering predetermined, evidence-based questions; second-year score correlation with in-training examination score; and satisfaction survey.ResultsA total of 76% passed scenario 1 (acute continuous RRT): 92% prescribed a ≥20 ml/kg per hour effluent dose; 63% estimated clearance as effluent volume. Forty-two percent passed scenario 2 (maintenance dialysis initiation); 75% correctly prescribed 3–4 mEq/L K+ dialysate and 12% identified the two absolute, urgent indications for maintenance dialysis initiation (uremic encephalopathy and pericarditis). Six percent passed scenario 3 (acute life-threatening hyperkalemia); 20% checked for rebound hyperkalemia with two separate blood draws. Eighty-three percent correctly withheld intravenous sodium bicarbonate for acute hyperkalemia in a nonacidotic, volume-overloaded patient on maintenance dialysis, and 32% passed overall. Second-year versus first-year fellow overall score was 44.4±4 versus 42.7±5 (one-tailed P=0.02), with 39% versus 24% passing (P=0.08). Second-year in-training examination and OSCE scores were not significantly correlated (r=0.15; P=0.26). Seventy-seven percent of fellows agreed the OSCE was useful in assessing “proficiency in ordering” acute KRT. Limitations include lack of a validated criterion test, and unfamiliarity with open-ended question format.ConclusionsThe OSCE can provide quantitative data for formative Accreditation Council for Graduate Medical Education competency assessments and identify opportunities for dialysis curriculum development.PodcastThis article contains a podcast at https://www.asn-online.org/media/podcast/CJASN/2019_08_08_CJASNPodcast_19_09_.mp3

show abstract

“…The practice of using formative testing tools, such as standardized in-training examinations, is integrated across graduate medical education programming as a means of resident and program assessment. In addition, performance on these examinations have demonstrated varying degrees of correlation to future board certification examinations in a host of physician programs such as internal medicine, family medicine, pediatrics,infectious disease, rheumatology, cardiology, nephrology, and several others 9,10,[15][16][17][18][19][20][21]. The framework for this study, CBAM, was created as a way for education specialists to understand how teachers were implementing new innovations and strategies into the classroom and to research how the successes demonstrated in educational research may not carry into practice; the fidelity -or adherence to the original plan-was lost when transitioning from a controlled environment to real-world practice 22,23.…”

mentioning

confidence: 99%

Pharmacy residency directors' concerns with implementing in‐training examinations in pharmacy residencies

Longyhore

2020

J Am Coll Clin Pharm

View full text Add to dashboard Cite

IntroductionThe use of in‐training examinations as formative assessment in graduate medical education is commonplace and performance has varying degrees of positive correlation with physician performance on future board certification examinations. Such a formative tool does not exist in graduate pharmacy education and may be a useful tool to consider developing and implementing into pharmacy residency programming.ObjectivesTo evaluate the concerns of residency program directors (RPDs) should they be faced with implementing an in‐training examination into their postgraduate year 1 (PGY1) pharmacy residency program.MethodsThe concerns‐based adoption model framework was used to conduct a mixed‐methods evaluation of RPDs' concerns about in‐training examinations in pharmacy residency training. RPDs from the eastern United States were invited to participate in a 35‐item survey, followed by a scripted interview to learn about their concerns with the idea of examination integration.ResultsTwenty‐four program directors completed the stages of concern (SOC) quantitative survey and seven continued on to complete the levels of use qualitative interview. Survey data identified that program directors were in the informational stage, requiring more information and understanding before implementing such an assessment tool. The qualification of their survey responses contained themes of equality and equity between participating programs, the difference between summative and formative assessment, and curriculum changes that would accompany such a programmatic change. All participants were in the early SOC and had not actively sought information about this practice before being part of this research.ConclusionSignificant efforts will be required to inform RPDs about in‐training examinations and formative assessment practices leading up to the implementation of such a practice in graduate pharmacy education.

show abstract

Performance on the Nephrology In-Training Examination and ABIM Nephrology Certification Examination Outcomes

Abstract: American Society of Nephrology In-Training Examination performance is significantly associated with American Board of Internal Medicine Nephrology Certification Examination score and passing status.

Cited by 15 publications

References 20 publications

A Systematic Review of the Relationship Between In-Training Examination Scores and Specialty Board Examination Scores

A Systematic Review of the Relationship Between In-Training Examination Scores and Specialty Board Examination Scores

The Acute Dialysis Orders Objective Structured Clinical Examination (OSCE)

Pharmacy residency directors' concerns with implementing in‐training examinations in pharmacy residencies

Contact Info

Product

Resources

About