Assessing professionalism in the context of an objective structured clinical examination: an in-depth study of the rating process

Abstract

This study highlights the complexity of the processes involved in assigning ratings to doctor-patient encounters. Greater emphasis on behavioural definitions of specific behaviours may not be a sufficient solution, as raters appear to vary in both attention to and evaluation of behaviours. Reliance on global ratings is also problematic, especially if relatively few raters are used, for similar reasons. We propose a model highlighting the multiple points where raters viewing the same encounter may diverge, resulting in different ratings of the same performance. Progress in assessment of professionalism will require further dialogue about what constitutes professional behaviour in the medical encounter, with input from multiple constituencies and multiple representatives within each constituency.

An Investigation of the Standardised Patient Interview Rating Scale (SPIRS) for the Assessment of Speech Pathology Students in a Simulation Clinic

“…Crossley and Jolly (2012) suggested that raters can disagree on the interpretation of a rating scale, even when they agree on their observations. Furthermore, raters of a similar background can focus on different aspects of an interview they are watching, with contrasting results (Mazor et al 2007). Mazor et al (2007), in their study of assessment of medical students' professionalism in an OSCE situation, found that different raters evaluated professionalism by considering a range of contrasting behaviours, resulting in diverse rating outcomes.…”

“…Furthermore, raters of a similar background can focus on different aspects of an interview they are watching, with contrasting results (Mazor et al 2007). Mazor et al (2007), in their study of assessment of medical students' professionalism in an OSCE situation, found that different raters evaluated professionalism by considering a range of contrasting behaviours, resulting in diverse rating outcomes. In the current study, shared discussion about both the scale and the observations therefore may be valuable.…”

The mini Clinical Evaluation Exercise (mini-CEX) in a pre-registration osteopathy program: Exploring aspects of its validity

“…35 This R-squared value was acceptable as Pell et al 35 suggests that correlations of greater than 0.5 indicate a good relationship. In accounting for the remaining variance of 12%, assessors may be using their own professional judgement biases to make their global rating decision 44 and/or making a judgement relative to other students at the same level of training. 29 These are entirely acceptable results as examiners are used on the basis of their professional background and competency.…”

Metodologias Ativas em Fisioterapia: Estudo de Confiabilidade Interexaminador do Método Osce

“…Nos casos clínicos contidos em cada estação, diversas temáticas podem ser exploradas, como executar um procedimento (anotações breves sobre história da doença atual, achados no exame físico, conduta terapêutica, por exemplo), dar ao paciente uma notícia ruim, observar o comportamento perante diferenças culturais, postura ética e até a linguagem no atendimento do paciente 1,9 . O Osce é parte integrante do processo avaliativo de muitos programas de residência médica no exterior e de exames de certificação profissional 10,11 .…”