Standardised clients as assessors in a veterinary communication OSCE: a reliability and validity study

Abstract

In human medicine, standardised patients (SP) have been shown to reliably and accurately assess learners' communication performance in high-stakes certification Objective Structured Clinical Examinations (OSCE), offering a feasible way to reduce the need for recruitment, time commitment and coordination of faculty assessors. In this study, we evaluated the use of standardised clients (SC) as a viable option for assessing veterinary students' communication performance. We designed a four-station, two-track communication skills OSCE. SC assessors used an adapted nine-item Liverpool Undergraduate Communication Assessment Scale (LUCAS). Faculty used a 21-item checklist derived from the Calgary-Cambridge Guide (CCG) and a five-point global rating scale. Participants were second year veterinary students (n=96). For the four stations, intrastation reliability (α) ranged from 0.63 to 0.82 for the LUCAS, and 0.73 to 0.87 for the CCG. The interstation reliability coefficients were 0.85 for the LUCAS and 0.89 for the CGG. The calculated Generalisability (G) coefficients were 0.62 for the LUCAS and 0.60 for the CGG. Supporting construct validity, SC and faculty assessors showed a significant correlation between the LUCAS and CCG total percent scores (r=0.45, P<0.001), and likewise between the LUCAS and global rating scores (r=0.49, P<0.001).Study results support that SC assessors offer a reliable and valid approach for assessing veterinary communication OSCE.

Assessing Communications Competencies through Reviews of Client Interactions and Comprehensive Rotation Assessment: A Comparison of Methods

“…In one study comparing scores of faculty members with those of actors trained as standardized clients and trained in use of a specific assessment tool, no difference in scores was identified. 23 In a study evaluating whether background of evaluators of OSCEs affected their scoring, there were significant differences in scores assigned by clinical science faculty members, basic science faculty members, faculty members who teach clinical communications, and hospital administrators/technicians, with those faculty who teach clinical communications assigning lower scores. 24 In the present study, it was also the clinical communications instructor who consistently scored students the lowest and it was the social worker, one of whose primary roles is to provide student support, who rated students the highest.…”

An integrated review of the role of communication in veterinary clinical practice

“…Although no study has yet reported the same benefits of web-based learning in veterinary medicine, this approach is expected to have the same potential in veterinary communication education. Use of virtual patients is considered as an effective way to assess students' communication skills in human medicine and as a potential communication assessment tool in veterinary medicine [2].…”

Leading a Clinical Communication Program in a Remote Setting: Practical Considerations from ROSS University School of Veterinary Medicine, St. Kitts

“…For example, we acknowledge that when we developed and implemented our first communication OSCE, there were concerns with surrounding noise and lack of a private and individualized space for students to demonstrate their skills in privacy. Though the setting was not ideal, it was functional and cost effective (Artemiou et al, 2013;Artemiou et al, 2014).…”

“…We purposely used a 21-item rubric that was based on the Calgary Cambridge Guide (CCG) and an adapted form of the Liverpool Undergraduate Communication Assessment Scale (LUCAS); both instruments are validated in human medicine to evaluate students' communication skills and we adapted these for veterinary communication OSCEs (Artemiou et al, 2013). Likewise, we also implement the validated approach in human medicine of engaging standardized clients in portraying their assigned roles as well as being trained in the role of the assessor for our veterinary communication OSCE (Artemiou et al, 2014).…”

“…Finally, in regards to assessment, the Objective Structured Clinical Examination (OSCE) offers a standardized and objective simulated environment that can be deliberately designed to achieve high validity and reliability to evaluate students' communication competence (Artemiou et al, 2014;Artemiou, Hecker, Adams, & Coe, 2015;Brannick, Erol-Korkmaz, & Prewett, 2011;. Running communication OSCEs can have several implications for a small SC team requiring that SCs are cross-trained to portray several OSCE cases, and in addition, our large class sizes often result in running a four to five-hour OSCE that poses additional challenges surrounding SC burnout, fatigue and reliability.…”