Is There Variability in Scoring of Student Surgical OSCE Performance Based on Examiner Experience and Expertise?

Donohoe, Claire L.; Reilly, Frank; Donnelly, Suzanne; Ra, Chisei

doi:10.1016/j.jsurg.2020.03.009

Cited by 11 publications

(7 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The greater seniority and rater experience of faculty physicians relative to senior trainees may explain why faculty physicians gave lower EPA scores in our study. However, conflicting literature shows that an assessor’s rater experience and trainee status do not influence such scores [ 19 , 20 ]. Some studies show that medical students or residents gave lower ratings than faculty physicians when evaluating their peers [ 14 , 15 ].…”

Section: Discussionmentioning

confidence: 99%

Comparing Entrustable Professional Activity Scores Given by Faculty Physicians and Senior Trainees to First-Year Residents

Katz

Wang

2022

Cureus

View full text Add to dashboard Cite

Section: Discussionmentioning

confidence: 99%

Comparing Entrustable Professional Activity Scores Given by Faculty Physicians and Senior Trainees to First-Year Residents

Katz

Wang

2022

Cureus

View full text Add to dashboard Cite

“…However, owing to faculty oversight, we do not consider this aspect to be a limitation (28). Furthermore, a recent study by Donohoe et al has indicated that neither clinical experience nor relevant content experience independently predicted the overall grade assigned in the OSCE (29). Finally, we used a thorough (long) checklist for our OSCE.…”

Section: Discussionmentioning

confidence: 99%

Effects of differential time and difficulty weighting on the improvement of OSCE quality metrics

Mihevc

Masnik²,

Petreski

et al. 2022

AMB

View full text Add to dashboard Cite

Namen: Objektivni strukturirani klinični izpit (OSKI) je postal vodilna metoda za ocenjevanje izvedbe kliničnih veščin. Njegova pogosta težava je neskladje med oceno na ocenjevalnem obrazcu in splošno oceno ocenjevalca. V preteklosti neskladij niso uspešno reševali z obteževanjem posameznih korakov. Sistematičnega diferencialnega obteževanja, ki podeljuje točke glede na izračun časovnih in težavnostnih uteži, še niso raziskali. Namen študije je bil: a) tvoriti formule za izračun težavnostnih in časovnih uteži, b) tvoriti eksperimentalni obrazec z upoštevanjem izračunanih uteži in c) oceniti uspešnost metode pri izboljšanju kazalnikov kakovosti OSKI. Metode: Najprej smo tvorili formule za izračun časovnih in težavnostih uteži posameznih korakov. Nato smo s pomočjo uteži obtežili posamezni korak v eksperimentalnem obrazcu. Kontrolni obrazec so neodvisno od raziskave obtežili klinični mentorji. Uspešnost obrazcev smo preverili med rednim izpitom OSKI za študente 3. letnika. Študente (n = 55) je z obema ocenjevalnima obrazcema in splošnim obrazcem ocenilo deset ocenjevalcev. Rezultate smo analizirali s t-testom za odvisna vzorca, enostavno linearno regresijo in psihometričnim testiranjem. Rezultati: V primerjavi s kontrolnim obrazcem smo na eksperimentalnem obrazcu ugotovili višjo povezavo med oceno na ocenjevalnem obrazcu in splošno oceno (r = 0,622, p < 0,001 vs. r = 0,496, p < 0,001) in pomembno nižje rezultate (p < 0,001). Zaključek: Ugotovili smo, da diferencialno obteževanje korakov s časovnimi in težavnostnimi utežmi lahko izboljša kazalnike kakovosti OSKI in zmanjša neskladje med oceno na ocenjevalnem obrazcu in splošno oceno.

show abstract

“…Assessor bias may also affect examination results and, in most studies, examiner variation is the single most significant component variable. Donohoe et al conducted a large-scale study to determine the role of clinical experience and subject matter expertise of assessors during an OSCE focused on surgical procedures [ 4 ]. The multivariate linear regression study found no significant correlation between score dispersion and either a surgeon or sub-specialist experience or consultant versus trainee levels of clinical competency.…”

Section: Reviewmentioning

confidence: 99%

“…In addition, as the medical practice becomes more subspecialized, clinical examiners may lack specific domain competence in every field and may not be equipped to appraise the current practice standards or student experiences in many subspecialties. Despite standardized checklists, the question remains as to whether each individual faculty member appreciates what they are observing as well as how their ability to rate students regardless of standardized checklists [ 4 ]. However, faculty observation of medical trainees is an essential component of assessment across specialties [ 5 ].…”

Section: Introductionmentioning

confidence: 99%

The Role of Videotaped Clinical Skills Aggregated Peer Evaluations in the Enhancement of Evaluation Skills of Individual Medical School Faculty Members

Duran¹,

Hendrix²

2023

Cureus

View full text Add to dashboard Cite

The conventional methodology for appraising medical students has some limitations like inherent subjectivity, unstructured nature, and bias. Implementing the Objective Structured Clinical Examination (OSCE) can mitigate these shortcomings. However, the OSCE presents challenges, including substantial financial costs and time-intensive processes, particularly when assessing a large cohort of students. Consequently, an alternative assessment was needed to keep the advantages of OSCE and mask its limitations, especially in resource-constrained settings. In addition, many scholars have expressed concern over medical students' and interns' inadequate interviewing and physical examination competencies in recent years. Due to easy availability, videotaping is a convenient method for objectively observing students for aggregate review by faculty in order to ascertain what exactly faculty are assessing during medical student evaluations. This technique allows for aggregate faculty group improvement in the ability of educators to assess students' technical proficiency, data collection capabilities, standardized patient interaction demeanor, and strategies for fostering standardized patient comfort. Nonetheless, aggregate evaluation of videotape recordings for faculty assessment development or reliance on verbal feedback from medical students about faculty's ability to assess student skill is a matter of debate. Due to increased subspecialization, the subspecialist or specialist examiners face difficulties in assessing students' skills in specific and/or generalized domains. Despite these challenges, assessing the ability of faculty members' observation and subsequent evaluation of medical trainees remains a vital aspect of assessment throughout various specialties. This paper presents the concept of faculty members individually observing and rating premade recordings of standardized students performing clinical skills for evaluation so that when the individual faculty members' ratings are aggregated and summarized collectively for evaluation by the faculty members as a group, both the group and the individual faculty members will gain a greater understanding of what are appropriate ratings versus outlier ratings.

show abstract

Is There Variability in Scoring of Student Surgical OSCE Performance Based on Examiner Experience and Expertise?

Cited by 11 publications

References 23 publications

Comparing Entrustable Professional Activity Scores Given by Faculty Physicians and Senior Trainees to First-Year Residents

Comparing Entrustable Professional Activity Scores Given by Faculty Physicians and Senior Trainees to First-Year Residents

Effects of differential time and difficulty weighting on the improvement of OSCE quality metrics

The Role of Videotaped Clinical Skills Aggregated Peer Evaluations in the Enhancement of Evaluation Skills of Individual Medical School Faculty Members

Contact Info

Product

Resources

About