Examining the validity argument for the Ottawa Surgical Competency Operating Room Evaluation (OSCORE): a systematic review and narrative synthesis

Spencer, Martha; Sherbino, Jonathan; Hatala, Rose

doi:10.1007/s10459-022-10114-w

Cited by 5 publications

(4 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, although our study found that the O‐EDShOT was well aligned with making decisions about learner progression and that it provided useful data to support those decisions, we did not directly assess these decisions or their impacts. There is a paucity of evidence for the implications inference for the O‐EDShOT, which mirror's the state of validity evidence for the O‐SCORE in general 32 . Future studies are needed to further explore the implications of the O‐EDShOT and to gather evidence for this inference.…”

Section: Discussionmentioning

confidence: 99%

“…There is a paucity of evidence for the implications inference for the O-EDShOT, which mirror's the state of validity evidence for the O-SCORE in general. 32 Future studies are needed to further explore the implications of the O-EDShOT and to gather evidence for this inference.…”

Section: Effect Of Individual Assessorsmentioning

confidence: 99%

“…27 It is based on the O-SCORE, 26 which is a retrospective entrustment-supervision scale 20,29 that has been widely adopted and adapted in other settings. [30][31][32][33] The purpose of the O-EDShOT is aligned with entrustment-supervision scales, because independently running an ED shift is a goal that trainees will work towards-usually over many years-and is a core component of readiness for independent practice. 27…”

Section: Ottawa Emergency Department Shift Observation Tool (O-edshot)mentioning

confidence: 99%

“…The O‐EDShOT (see Appendix A in Supporting Information) is an end‐of‐shift WBA that was developed to assess a trainee's readiness to assume an attending's role in independently running all aspects of an emergency department (ED) shift 27 . It is based on the O‐SCORE, 26 which is a retrospective entrustment‐supervision scale 20,29 that has been widely adopted and adapted in other settings 30–33 . The purpose of the O‐EDShOT is aligned with entrustment‐supervision scales, because independently running an ED shift is a goal that trainees will work towards—usually over many years—and is a core component of readiness for independent practice 27 …”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Assessing the utility of a novel entrustment‐supervision assessment tool

et al. 2023

View full text Add to dashboard Cite

BackgroundWork‐based assessments (WBAs) are increasingly used to inform decisions about trainee progression. Unfortunately, WBAs often fail to discriminate between trainees of differing abilities and have poor reliability. Entrustment‐supervision scales may improve WBA performance, but there is a paucity of literature directly comparing them to traditional WBA tools.MethodsThe Ottawa Emergency Department Shift Observation Tool (O‐EDShOT) is a previously published WBA tool employing an entrustment‐supervision scale with strong validity evidence. This pre‐/post‐implementation study compares the performance of the O‐EDShOT with that of a traditional WBA tool using norm‐based anchors.All assessments completed in 12‐month periods before and after implementing the O‐EDShOT were collected, and generalisability analysis was conducted with year of training, trainees within year and forms within trainee as nested factors. Secondary analysis included assessor as a factor.ResultsA total of 3908 and 3679 assessments were completed by 99 and 116 assessors, for 152 and 138 trainees in the pre‐ and post‐implementation phases respectively. The O‐EDShOT generated a wider range of awarded scores than the traditional WBA, and mean scores increased more with increasing level of training (0.32 vs. 0.14 points per year, p = 0.01). A significantly greater proportion of overall score variability was attributable to trainees using the O‐EDShOT (59%) compared with the traditional tool (21%, p < 0.001). Assessors contributed less to overall score variability for the O‐EDShOT than for the traditional WBA (16% vs. 37%). Moreover, the O‐EDShOT required fewer completed assessments than the traditional tool (27 vs. 51) for a reliability of 0.8.ConclusionThe O‐EDShOT outperformed a traditional norm‐referenced WBA in discriminating between trainees and required fewer assessments to generate a reliable estimate of trainee performance. More broadly, this study adds to the body of literature suggesting that entrustment‐supervision scales generate more useful and reliable assessments in a variety of clinical settings.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Effect Of Individual Assessorsmentioning

confidence: 99%