Relationships between high-stakes clinical skills exam scores and program director global competency ratings of first-year pediatric residents

Abstract

BackgroundResponding to mandates from the Accreditation Council for Graduate Medical Education (ACGME) and American Osteopathic Association (AOA), residency programs have developed competency-based assessment tools. One such tool is the American College of Osteopathic Pediatricians (ACOP) program directors’ annual report. High-stakes clinical skills licensing examinations, such as the Comprehensive Osteopathic Medical Licensing Examination Level 2-Performance Evaluation (COMLEX-USA Level 2-PE), also assess competency in several clinical domains.ObjectiveThe purpose of this study is to investigate the relationships between program director competency ratings of first-year osteopathic residents in pediatrics and COMLEX-USA Level 2-PE scores from 2005 to 2009.MethodsThe sample included all 94 pediatric first-year residents who took COMLEX-USA Level 2-PE and whose training was reviewed by the ACOP for approval of training between 2005 and 2009. Program director competency ratings and COMLEX-USA Level 2-PE scores (domain and component) were merged and analyzed for relationships.ResultsBiomedical/biomechanical domain scores were positively correlated with overall program director competency ratings. Humanistic domain scores were not significantly correlated with overall program director competency ratings, but did show moderate correlation with ratings for interpersonal and communication skills. The six ACGME or seven AOA competencies assessed empirically by the ACOP program directors’ annual report could not be recovered by principal component analysis; instead, three factors were identified, accounting for 86% of the variance between competency ratings.DiscussionA few significant correlations were noted between COMLEX-USA Level 2-PE scores and program director competency ratings. Exploring relationships between different clinical skills assessments is inherently difficult because of the heterogeneity of tools used and overlap of constructs within the AOA and ACGME core competencies.

JAOA/AACOM. Evidence-Based Redesign of the COMLEX-USA Series

“…In addition to reviewing current research and national survey data, the Panel sponsored research projects that contributed to the practice analysis, including a survey of the clinical procedures taught during residency. Other research efforts investigated the relationship between competency ratings of residents and COMLEX-USA scores, 25,26 contributing convergent evidence of validity. 7 With the intention of adding broad stakeholder opinions to the practice analysis, the Panel sponsored a test development survey that was distributed to more than…”

The Use of COMLEX-USA and USMLE for Residency Applicant Selection

“…8 Studies show that performance on the COMLEX-USA series is related to performance in undergraduate universities 12 and in osteopathic medical schools, [13][14][15][16][17][18][19][20][21] and that it can reliably predict performance in both American Osteopathic Association (AOA)-approved and ACGME-accredited residency programs. [22][23][24][25] However, an increasing number of osteopathic medical students are electing to take the USMLE Step 1 examination. [26][27][28] The purpose of this study was to investigate the relationship between individual examination performances on the COMLEX-USA Level 1 and USMLE…”

Impact of Critical Care Medicine Training Programs' Palliative Care Education and Bedside Tools on ICU Use at the End of Life

“…Previous work has shown good correlation between program directors' perspectives and externally validated metrics. 19 To quantify evidence-based support to affect the patient encounter, we identified bedside tools published in the medical literature. 18,20,21 The identified tools included a family booklet, 22 a point-of-care communication card, 17 withdrawal of lifesustaining treatment protocol, 20 a bereavement brochure, 17 and a comfort measure order set.…”