Oral assessment and postgraduate medical examinations: establishing conditions for validity, reliability and fairness

Abstract

The purpose of this review was to examine the practice of oral assessment in postgraduate medical education in the context of the core assessment constructs of validity, reliability and fairness. Although oral assessment has a long history in the certification process of medical specialists and is a well-established part of such proceedings for a wide range of specialties in most countries, there remains concern regarding its use. Therefore, there has been some move away from oral assessment for postgraduate medical education in some countries. This review also highlights the complexity of oral assessment as an examination format, partly through a consideration of the six dimensions of oral assessment, and raises concerns about the validity, reliability and fairness of such an assessment procedure for the award of certification of completion of the specialist training. Supporting high quality published research into examination practices and outcomes and acting on the findings of such research is needed urgently to allay concerns about the transparency and fairness of these examinations, especially when assessing international medical graduates. The article concludes by proposing 15 conditions under which oral assessment is valid, reliable and fair. PMID: 18386152 [PubMed -indexed for MEDLINE]

Appropriateness of using oral examination as an assessment method in medical or dental education

“…[10] Flexibility in moving from one area to another during the examination is cited as an added advantage of oral exams. [11] Usefulness of oral exams Kehm' s review of the literature highlights potential challenges when using oral vivas as socio-physiological (unequal power distance between the candidate and the assessors), psycho-analytical (ritual aspects of the process), sociological (differences in cultural and social norms affecting oral performance), and methodological issues of measurement (low degree of objectivity, reliability, and validity). [12] The use of oral examinations in high-stakes assessment systems has also been criticized for many years because of poor inter rater reliability.…”

“…The reasons for low reliability also have an adverse impact on validity because of the potential for variation in content matter addressed, and in the emphasis given to different content areas. [11] Oral examinations are usually employed in an attempt to assess the candidate' s knowledge of a subject. Memon and his colleagues pointed out that viva marks correlated with personality scores.…”

“…Memon and his colleagues pointed out that viva marks correlated with personality scores. [11] Rowland-Morin et al, showed that verbal style and dress of the candidates influence oral examination scores. [13] Roberts et al carried out a discourse analysis (a detailed study of language in use) of the oral component in the membership examination of the Royal College of General Practitioners (MRCGP), and pointed out that candidates from ethnic minorities and those trained abroad may experience particular hidden difficulties with oral examinations leading to discrimination.…”

Selecting general practice specialty trainees: where next?

“…This finding is consistent with other postgraduate assessment across all specialties. 16 While to an extent it may not be surprising that such a doctor might be unfamiliar with British culture, medical practices, and the peculiarities of the NHS, it is incumbent on those delivering such assessments to prove that they remain truly equitable and unbiased.…”

A new form of assessment for paediatric trainees: readiness for consultant practice

“…The strengths of oral assessment include the ability to probe the depth of knowledge, to test clinical decision-making skills and to recognise safe and competent clinicians 17. However, oral assessments have been criticised as being unreliable, invalid, biased and open to challenge 18 19. There are concerns that personality,20 verbal style and appearance21 22 influence assessment scores and thus challenge validity.…”