Differential Weighting for Subcomponent Measures of Integrated Clinical Encounter Scores Based on the USMLE Step 2 CS Examination: Effects on Composite Score Reliability and Pass–Fail Decisions

Park, Yoon Soo; Lineberry, Matthew; Hyderi, Abbas; Bordage, Georges; Xing, Kuan; Yudkowsky, Rachel

doi:10.1097/acm.0000000000001359

Cited by 17 publications

(31 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another solution that could be used is to decrease the weight on the less reliable component; however, this could result in bias since the assessment of some critical thinking and reasoning skills, assessed by OSPE, could be undermined by this approach. Despite improvements in composite score reliability, as seen in a similar investigation, the low reliability of OSPE has alerted investigators to identify methods to improve processes in developing, administering, and scoring the OSPE in subsequent test administrations. Efforts are also underway to evaluate the inter‐rater reliability of the OSPE.…”

Section: Discussionmentioning

confidence: 95%

“…The original Kane and Case approach describes creating composite scores for 2 subscores. Recently, Park et al published an extension for multiple scores (more than 2), which is used here with the dental assessment data.…”

Section: Methodsmentioning

confidence: 99%

“…A different approach that can provide a more predictable take on combining scores is the composite scoring approach. Kane and Case proposed that by incorporating the correlation between elements as well as the reliability of individual elements, we could generate a more reliable grade that is not skewed by 1 particular component, thereby generating a more reproducible score . The Kane and Case approach has been applied to medical assessments and has shown to improve the precision of composite measures—this approach can serve as a useful method in dental education to improve the reliability and validity of assessment scores.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Comparison of weighted and composite scores for pre‐clinical dental learners

Nassar

Park²,

Tekian³

2017

Eur J Dental Education

View full text Add to dashboard Cite

show abstract

Section: Discussionmentioning

confidence: 95%

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Comparison of weighted and composite scores for pre‐clinical dental learners

Nassar

Park²,

Tekian³

2017

Eur J Dental Education

View full text Add to dashboard Cite

show abstract

“…Each SP encounter was limited to 15 minutes and was followed by 10 minutes in which the student was expected to write the PN. Validity studies of the GCE using Messick's unified validity framework have been reported previously, [23][24][25] indicating consistency in scores and validity evidence with regard to content, response process, internal structure, relationships to other variables, and consequences.…”

Section: Graduation Competency Examinationmentioning

confidence: 81%

Pre‐medical majors in the humanities and social sciences: impact on communication skills and specialty choice

2018

Self Cite

View full text Add to dashboard Cite

Context Medical school admissions committees use a variety of criteria to determine which candidates to admit to their programmes. Effective communication is increasingly considered a key requisite to the practice of effective medicine. Medical students with pre‐medical backgrounds in the humanities and social sciences may be more likely to acquire skills relevant to patient‐centred communication, either prior to or during medical school. Objectives The purpose of this study was to investigate the relationship between pre‐medical backgrounds in the humanities and social sciences and outcomes in medical school, including in communication and interpersonal skills (CIS), licensure examination results and postgraduate specialty choice (primary care versus non‐primary care specialties). Methods The American Medical College Application Service database was used to identify pre‐medical college majors, demographic characteristics, Medical College Admission Test scores and college grade point averages for medical students at a large, midwestern medical school. Data were obtained for 465 medical students across three cohorts (classes of 2014–2016). Correlation and regression analyses were used to examine relationships between pre‐medical background, performance on graduation competency examination standardised patient encounter CIS scores and on United States Medical Licensing Examination (USMLE) Step 1 and Step 2 Clinical Knowledge scores, and postgraduate specialty choice. Results Graduating medical students with pre‐medical humanities or social sciences majors performed significantly better in terms of CIS than those with natural science majors (Cohen's d = 0.28, p = 0.011). There were no significant associations between pre‐medical majors and USMLE Step 1 and Step 2 Clinical Knowledge scores or postgraduate specialty choice. Conclusions These results suggest that considering humanistic factors as part of admissions criteria may promote the selection and training of physicians with good communication skills.

show abstract

“…Traditional, 5- to 10-station OSCEs are time-consuming and expensive. 2,6,7 This study describes the reliability of a more feasible 2-station clerkship OSCE at 1 institution for 6 clerkships. The data are interesting in that the reliability for some clerkships (internal medicine, pediatrics, surgery) appears poor while the reliability for other (obstetrics and gynecology, primary care, psychiatry) appears fair.…”

Section: Discussionmentioning

confidence: 99%

The Reliability of 2-Station Clerkship Objective Structured Clinical Examinations in Isolation and in Aggregate

Bernard

Feinn

Ceccolini

et al. 2019

Journal of Medical Education and Curricular Development

View full text Add to dashboard Cite

Background: Most medical schools in the United States report having a 5- to 10-station objective structured clinical examination (OSCE) at the end of the core clerkship phase of the curriculum to assess clinical skills. We set out to investigate an alternative OSCE structure in which each clerkship has a 2-station OSCE. This study looked to determine the reliability of clerkship OSCEs in isolation to inform composite clerkship grading, as well as the reliability in aggregate, as a potential alternative to an end-of-third-year examination. Design: Clerkship OSCE data from the 2017-2018 academic year were analyzed: the generalizability coefficient (ρ2) and index of dependability (φ) were calculated for clerkships in isolation and in aggregate using variance components analysis. Results: In all, 93 students completed all examinations. The average generalizability coefficient for the individual clerkships was .47. Most often, the largest variance component was the interaction between the student and the station, indicating inconsistency in the performance of students between the 2 stations. Aggregate clerkship OSCE analysis demonstrated good reliability for consistency (ρ2 = .80). About one-third (33.8%) of the variance can be attributed to students, 8.2% can be attributed to the student by clerkship interaction, and 42.6% can be attributed to the student by block interaction, indicating that students’ relative performances varied by block. Conclusions: Two-station clerkship OSCEs have poor to fair reliability, and this should inform the weighting of the composite clerkship grade. Aggregating data results in good reliability. The largest source of variance in the aggregate was student by block, suggesting testing over several blocks may have advantages compared with a single day examination.

show abstract

Differential Weighting for Subcomponent Measures of Integrated Clinical Encounter Scores Based on the USMLE Step 2 CS Examination: Effects on Composite Score Reliability and Pass–Fail Decisions

Abstract: Medical schools and certification agencies should consider implications of assigning weights with respect to composite score reliability and consequences on pass-fail decisions.

Cited by 17 publications

References 21 publications

Comparison of weighted and composite scores for pre‐clinical dental learners

Comparison of weighted and composite scores for pre‐clinical dental learners

Pre‐medical majors in the humanities and social sciences: impact on communication skills and specialty choice

The Reliability of 2-Station Clerkship Objective Structured Clinical Examinations in Isolation and in Aggregate

Contact Info

Product

Resources

About