Evaluating the validity evidence of an OSCE: results from a new medical school

Abstract

BackgroundTo prevent the problems of traditional clinical evaluation, the “Objective Structured Clinical Examination (OSCE)” was presented by Harden as a more valid and reliable assessment instrument. However, an essential condition to guarantee a high-quality and effective OSCE is the assurance of evidence to support the validity of its scores. This study examines the psychometric properties of OSCE scores, with an emphasis on consequential and internal structure validity evidence.MethodsFifty-three first year medical students took part in a summative OSCE at the Lebanese American University-School of Medicine. Evidence to support consequential validity was gathered by using criterion-based standard setting methods. Internal structure validity evidence was gathered by examining various psychometric measures both at the station level and across the complete OSCE.ResultsCompared to our actual method of computing results, the introduction of standard setting resulted in lower students’ average grades and a higher cut score. Across stations, Cronbach’s alpha was moderately low.ConclusionGathering consequential and internal structure validity evidence by multiple metrics provides support for or against the quality of an OSCE. It is critical that this analysis be performed routinely on local iterations of given tests, and the results used to enhance the quality of assessment.Electronic supplementary materialThe online version of this article (10.1186/s12909-018-1421-x) contains supplementary material, which is available to authorized users.

The 3-D Skills Model: a Randomised Controlled Pilot Study Comparing a Novel 1–1 Near-Peer Teaching Model to a Formative OSCE with Self-regulated Practice

“…In addition, as the control repeated the final station, we have a measure of test–retest reliability. These measures address validity and reliability of our OSCE assessment and evidence consideration of Messick’s validity framework, seen as a standard when assessing the evidence to support a test [ 24 , 25 ].…”

It matters how we do it: A review of best practices of Observed Structured Clinical Examination in pharmacy education

“…As per the developed blueprint, all stations are primarily developed by the pharmacy school's OSCE team which should include the lead and the pharmacy academic staff, with the input from the subject matter experts (Khan et al, 2013). The blueprint should then be periodically reviewed to ensure the internal structure validity by closely examining the psychometric measures encompassing inter-item correlations, examination difficulty, and score reliability (Yazbeck Karam et al, 2018). The OSCE lead also has the overall responsibility of managing the development, organisation, and conducting of the OSCE (Mccoy & Merrick, 2001).…”

“…These include: the evidence to show that the assessment tool has been mapped to the curriculum (blueprinting), the evidence to demonstrate that it accurately records students' responses to the items being assessed (checklists), the evidence to show that the tool is reliable; and the evidence to demonstrate that the consequences of passing or failing the examinations are justified (Khan & Ramachandran, 2012). The validity of OSCE stations is established by using criterion-based standard-setting methods as described above as well as by examining the psychometric measures both at the station level and across the entire OSCE (Yazbeck et al, 2018).…”

Percepciones de la evaluación clínica objetiva estructurada en profesores y estudiantes de una universidad colombiana

“…Cada competencia puede desagregarse; así, en las asistenciales se pueden evaluar la capacidad para recolectar información del paciente, diligenciamiento de la historia clínica y la exploración física, elaboración de un informe clínico, comunicación eficaz con el paciente y sus familiares, conocimiento y abordaje de una patología, interpretación de pruebas complementarias, jerarquización de problemas clínicos, realización de diagnósticos diferenciales, aplicación de planes terapéuticos; incluso habilidades técnicas como tomar la presión sanguínea o hacer una sutura(Daniels & Pugh, 2018;Lafleur et al, 2015;. Saeed et al, 2012;Stansfield et al, 2016;Yazbeck et al, 2018).En este último ámbito puede situarse la ECOE, en la cual se diseña una serie de casos clínicos para evaluar diferentes competencias. La validez y fiabilidad de ECOE debe reunir una serie de características básicas, como una duración entre tres y cuatro horas, tener ocho o más pacientes simulados, cada estación debe durar diez minutos, tener un máximo de 30 ítems de evaluación por caso, no reunir más de 20 candidatos a evaluar en cada sesión y combinar varios instrumentos(Battistone et al, 2017;Daniels & Pugh, 2018;Saeed et al, 2012;Yazbeck et al, 2018).…”

“…Saeed et al, 2012;Stansfield et al, 2016;Yazbeck et al, 2018).En este último ámbito puede situarse la ECOE, en la cual se diseña una serie de casos clínicos para evaluar diferentes competencias. La validez y fiabilidad de ECOE debe reunir una serie de características básicas, como una duración entre tres y cuatro horas, tener ocho o más pacientes simulados, cada estación debe durar diez minutos, tener un máximo de 30 ítems de evaluación por caso, no reunir más de 20 candidatos a evaluar en cada sesión y combinar varios instrumentos(Battistone et al, 2017;Daniels & Pugh, 2018;Saeed et al, 2012;Yazbeck et al, 2018).…”