Validity evidence for two objective structured clinical examination stations to evaluate core skills of the shoulder and knee assessment

Abstract

BackgroundWe developed two objective structured clinical examinations (OSCEs) to educate and evaluate trainees in the evaluation and management of shoulder and knee pain. Our objective was to examine the evidence for validity of these OSCEs.MethodsA multidisciplinary team of content experts developed checklists of exam maneuvers and criteria to guide rater observations. Content was proposed by faculty, supplemented by literature review, and finalized using a Delphi process. One faculty simulated the patient, another rated examinee performance. Two faculty independently rated a portion of cases. Percent agreement was calculated and Cohen’s kappa corrected for chance agreement on binary outcomes. Examinees’ self-assessment was explored by written surveys. Responses were stratified into 3 categories and compared with similarly stratified OSCE scores using Pearson’s coefficient.ResultsA multi-disciplinary cohort of 69 examinees participated. Examinees correctly identified rotator cuff and meniscal disease 88% and 89% of the time, respectively. Inter-rater agreement was moderate for the knee (87%; k = 0.61) and near perfect for the shoulder (97%; k = 0.88). No correlation between stratified self-assessment and OSCE scores were found for either shoulder (0.02) or knee (−0.07).ConclusionsValidity evidence supports the continuing use of these OSCEs in educational programs addressing the evaluation and management of shoulder and knee pain. Evidence for validity includes the systematic development of content, rigorous control of the response process, and demonstration of acceptable interrater agreement. Lack of correlation with self-assessment suggests that these OSCEs measure a construct different from learners’ self-confidence.Electronic supplementary materialThe online version of this article (doi:10.1186/s12909-016-0850-7) contains supplementary material, which is available to authorized users.

Stress, anxiety, self-efficacy, and the meanings that physical therapy students attribute to their experience with an objective structured clinical examination

“…In an OSCE, each student is required to demonstrate specific skills and behaviors in a simulated work environment. An OSCE typically consists of a series of assessment tasks (stations) simulating real-life clinical situations with an actor [4,8,9]. Students need to demonstrate their skills within a standard time limit.…”

Effectiveness of an Interprofessional and Multidisciplinary Musculoskeletal Training Program

“…Validity evidence of these has been reported; interrater agreement was 87% (k ¼ 0.61) and 97% (k ¼ 0.88) for the knee and shoulder, respectively. 25 This project was reviewed by the Institutional Review Board of the University of Utah and was classified as exempt.…”

Using Peer Role-playing to Improve Student Skills for Musculoskeletal Physical Examinations

“…Team teaching using an OSCE format would provide a more sustainable approach for clinical skills training in a structured program of medical education. Battistone used OSCE stations developed by two orthopaedic surgeons, two rheumatologists, and a primary care provider with orthopaedic experience [12]. Other studies have recommended the use of near-peer teaching in medical student education, particularly for hands-on skills [13].…”