Twelve tips for developing an OSCE that measures what you want

Daniels, Vijay; Pugh, Debra

doi:10.1080/0142159x.2017.1390214

Cited by 75 publications

(58 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…We have established that the use of videoing for the purpose of OSCE appeals is not currently supported by the literature, and nor by the majority of institutions running high‐stakes OSCEs across both undergraduate and postgraduate medical education in the UK. We therefore refer readers to the extensive literature on steps that can be taken to improve the reliability of OSCEs without resorting to the use of video recording . Adequate examiner training, clear marking descriptors, a sufficient number of OSCE stations and careful psychometric analysis all play a part.…”

Section: Resultsmentioning

confidence: 99%

Should we video OSCEs for student appeals?

Hogley

Thampy

Fisher

2019

The Clinical Teacher

View full text Add to dashboard Cite

Background Those setting objective structured clinical exams (OSCEs) are likely to encounter candidates who dispute the examiners’ scores and feedback. At our own institution, candidates have requested video recording to aid in later academic appeals. Before ceding to such requests, we wished to be certain that the marking of recorded OSCEs would give comparable results to live marking, and that technical challenges could be met within reasonable cost. Methods We adopted a mixed‐methods approach to explore the use of video recording OSCEs. Following an initial literature review, we surveyed and interviewed the assessment leads of UK medical schools and postgraduate institutions setting high stakes OSCEs to explore their views and experience in using recordings as part of summative OSCE assessments. Results Results from our literature review reveal inconsistent findings about the comparability of marks awarded from video marking and those awarded by examiner in‐station marking, with suggestions that video marking generates lower scores. The majority of UK medical education institutions were neither videoing OSCEs nor considering doing so. Many logistical and assessor‐related drawbacks to video review were identified, with significant concerns raised about how such a process could be feasibly managed. Discussion Although the videoing of OSCEs may superficially appear an easy solution to provide defensibility of the process, the potential use is beset with challenge, going beyond purely logistical and technical difficulties. The use of videoing for the purpose of OSCE appeals is not currently supported by the literature, nor by the majority of UK undergraduate and postgraduate medical education institutions delivering high‐stakes summative OSCEs.

show abstract

Section: Resultsmentioning

confidence: 99%

Should we video OSCEs for student appeals?

Hogley

Thampy

Fisher

2019

The Clinical Teacher

View full text Add to dashboard Cite

show abstract

“…The Objective Structured Clinical Examination (OSCE) (Harden, Stevenson, Downie, & Wilson, 1975) is an example of CBA that is used globally for both summative and formative purposes in medical education (V. J. Daniels & Pugh, 2017).…”

Section: Competency-based Assessment (Cba)mentioning

confidence: 99%

“…This has been the dominant reason for the widespread use of OSCEs in medical education since the OSCE was first introduced in 1975 (V. J. Daniels & Pugh, 2017;Patrício, Julião, Fareleira, & Carneiro, 2013).…”

Section: Objective Structured Clinical Examination (Osce) In Medical mentioning

confidence: 99%

See 1 more Smart Citation

Consistency of examiner judgements in medical education: a case study of competency-based assessment

Wong¹

View full text Add to dashboard Cite

Higher education institutions globally face decreasing government funding and heightened accountability. There is an increasing demand for them to justify the use of public funding for teaching in universities, and provide information for national and international comparisons of quality. High-stakes summative student assessment data have been, and continue to be, critical for fulfilling both accountability and quality requirements. However, concerns have been raised about the dominance of using summative assessment for multiple purposes beyond certifying student achievement. For example, this dominance marginalises the importance of formative assessment in providing feedback to guide students' future learning. Such concerns have driven assessment reforms. One of these reforms-competency-based assessment (CBA)-is promoted as an alternative to high-stakes examinations, as it may achieve both summative and formative purposes. In the context of medical education, an example of CBA is the Objective Structured Clinical Examination (OSCE) which has been implemented globally for both summative and formative purposes. The OSCE was originally designed for a small cohort of students (around 100) to undertake highly structured and discrete clinical tasks in a series of timed stations. A major challenge of the OSCE relates to achieving consistency of examiner judgements of student performance as the assessment format relies on multiple examiners. In this PhD study, the OSCE under investigation was a high-stakes exit examination for large cohorts of final-year students (over 350) enrolled in a graduate entry four-year Bachelor of Medicine/Bachelor of Surgery (MBBS) program at one Australian research-intensive university. A question arises as to whether examiners in this context can deliver consistent and reliable judgements. This is the issue of concern addressed in this thesis. The overarching purpose of this study was to provide new insights into the consistency of examiner judgements in this high-stakes assessment, and explore the possible impact of structured feedback on changing examiner marking behaviour. The four specific aims were to: develop a deeper understanding of the associated factors that influence examiner judgements of medical students' performance in clinical examinations (OSCEs); evaluate the impacts of providing examiners with structured feedback on their subsequent judgement behaviours; explore the factors that impact on the effectiveness of structured feedback in changing examiner marking behaviour; and, to explore the examiners' proposed training strategies that may assist in increasing the consistency of their judgements in OSCEs. A mixed-methods case study approach was adopted to collect both quantitative and qualitative data. Quantitative data included the examiners' scores awarded to the students in the Year 1 and 2 OSCEs. After completing the OSCE in Year 1, the examiners received a structured Consistency of Examiner Judgements in Medical Education 3 feedback report about their marking behaviour pr...

show abstract

“…Teórica e Prática sobre RCP noAdulto em SBV com o uso do DEA no Ambiente HospitalarOs efeitos no processo de ensino-aprendizagem surgem quando a própria avaliação exige a aprendizagem, impondo aos educadores uma visão para que as avaliações sejam implementadas com o objetivo de agregar valor pedagógico(PUGH et al, 2017), e, não só o caráter formativo e/ou somativo. Os métodos de avaliação aplicados devem inspirar a necessidade de aprendizagem(ABDULGHANI et al, 2015).As avaliações exercem forte influência sobre a forma de estudo adotada pelo aprendiz, por isso devem ser projetadas para fomentar desejáveis habilidades e raciocínio(LAFLEUR; COTÊ;LEPPINK, 2015;PUGH;DESJARDINS; EVA, 2017, SOBH et al, 2017.A avaliação desempenha um papel importante no encorajamento da aprendizagem e, simultaneamente, é identificado se os objetivos de aprendizagem estão sendo alcançados por meio do resultado da avaliação, além disso, identifica os pontos fortes e as fragilidades do processo de ensino-aprendizagem e possibilita o desenvolvimento de melhorias (COUGHLIN; FEATHERSTONE, 2017).É indispensável avaliar a aprendizagem por meio de instrumentos válidos e confiáveis.…”

unclassified

Objetos contemporâneos para ensino-aprendizagem da ressuscitação cardiopulmonar

Alves¹

View full text Add to dashboard Cite

Introdução: O processo de ensino-aprendizagem na contemporaneidade deve ser motivo de inquietude pelo avanço da tecnologia e pelo perfil dos nativos digitais. O uso de Tecnologias Educacionais Digitais (TED) deve ser inserido no ensino de Ressuscitação Cardiopulmonar (RCP). A American Heart Association (AHA) incentiva o ensino da RCP em diferentes modalidades. Objetivo: Desenvolver TED videoaula, vídeo de simulação, e instrumentos de avaliação-teórico e prático, sobre RCP no adulto em Suporte Básico de Vida (SBV), com o uso do Desfibrilador Externo Automático (DEA), no ambiente hospitalar. Metodologia: Trata-se de uma pesquisa aplicada e de produção tecnológica desenvolvida na Escola de Enfermagem de Ribeirão Preto da Universidade de São Paulo (EERP/USP). A população foi composta por 16 experts em Urgência e Emergência (UE). Para a avaliação e validação, por meio de instrumentos adaptados, do roteiro/script e storyboard de uma videoaula, roteiro/script e storyboard de um vídeo de simulação, questionário e Exame Clínico Objetivo e Estruturado (OSCE), sobre RCP no adulto em SBV com o uso do DEA em ambiente hospitalar. Para seleção dos expertises foi adotado pontuação mínima de cinco pontos nos critérios de Fehring (1987). Na trajetória metodológica para elaboração dos vídeos foi adotado o modelo proposto por Fleming, Reynolds e Wallace (2009) que consiste em Fase I: Pré-produção, Fase II: Produção e Fase III: Pós-produção. Para elaboração do questionário foi aplicado as regras básicas para elaboração de Questão de Múltipla Escolha (QME) do manual do Conselho Nacional de Examinadores Médicos. O OSCE foi elaborado de acordo com as orientações do Medical Concil of Canadá. As diretrizes da AHA publicadas em 2015 foram a primeira literatura adotada. A pesquisa foi aprovada no Comitê de Ética em Pesquisa (CEP), conforme Resolução 466/2012. Para a apreciação dos dados foi aplicado estatística descritiva e análise de concordância inter-avaliadores por AC1 de Gwet. Categorização da concordância de acordo com Landis e Koch (1997). Os vídeos foram validados em relação ao objetivo, conteúdo, relevância, ambiente, linguagem verbal e inclusão de tópicos. O questionário e OSCE foram validados em relação à organização, objetividade e clareza. Resultados: Os experts são compostos por enfermeiros (100%), predominância do sexo feminino, idade média de 36,56 anos, média de tempo de formação de 12,93 anos, 93,75% com titulação em mestrado, 93,75% com prática clínica em UE no adulto, distribuídos nas regiões Sudeste, Centro-Oeste e Nordeste do Brasil, 62,5% classificados nos critérios de Fehring (1987), com dez ou mais pontos. Todos os itens dos instrumentos de validação registraram predomínio em respostas positivas. Em relação à concordância inter-avaliadores a videoaula foi classificada em "concordância moderada", vídeo de simulação em "concordância considerável", questionário e OSCE em "concordância quase perfeita". A videoaula foi finalizada Descritores: Ensino. Aprendizagem. Tecnologia Educacional. Ressuscitação Cardiopulmonar. Sup...

show abstract

Twelve tips for developing an OSCE that measures what you want

Cited by 75 publications

References 41 publications

Should we video OSCEs for student appeals?

Should we video OSCEs for student appeals?

Consistency of examiner judgements in medical education: a case study of competency-based assessment

Objetos contemporâneos para ensino-aprendizagem da ressuscitação cardiopulmonar

Contact Info

Product

Resources

About