MRCGP CSA: are the examiners biased, favouring their own by sex, ethnicity, and degree source?

Abstract

BackgroundConcern exists regarding differential performance of candidates in postgraduate clinical assessments by ethnicity, sex, and country of primary qualification. Could examiner bias be responsible? AimTo explore whether candidate demographics affect examiners' judgements, by investigating candidates' case performances by candidates' and examiners' demographics. Design and settingData on 4000 candidates (52 000 cases) sitting the MRCGP clinical skills assessment in 2011-2012. MethodUnivariate analyses were undertaken of subgroup performance (male/female, white/ black and minority ethnic (BME), UK/non-UK graduates) by parallel examiner demographics. Due to confounding of variables, these were complemented by multivariate ANOVA and multiple regression analyses. ResultsUnivariate analysis showed some differences between outcomes between the same-group and other-group examiners: these were contradictory regarding examiners 'favouring their own', for example, males received higher marks from female examiners than from males: maximum effect size was 3.6%. A six-way ANOVA confirmed all three candidate and examiner variables as having significant effects individually, identifying one significant interaction (examiner sex by examiner ethnicity). Stepwise regression showed candidate variables predicting 12% of score variance, parallel examiner demographics adding little (approximately 0.2% of variance). One 'transactional' variable proved significant, explaining 0.06% of score variance. ConclusionExaminers show no general tendency to 'favour their own kind'. With confounding between variables, as far as the impact on candidates' case scores, substantial effects relate to candidate and not examiner characteristics. Candidate-examiner interaction effects were inconsistent in their direction and slight in their calculated impact.

PLAB and UK graduates' performance on MRCP(UK) and MRCGP examinations: data linkage study

“…A recommendation of the Esmail and Roberts report was that "further research should be commissioned … to investigate how black and minority ethnic standardised patients and black and minority ethnic examiners score candidate physicians who are racially and ethnically concordant and compare that to how non-concordant standardised patients and examiners score the black and minority ethnic candidates" (p 19). Group analyses of examiner and candidate concordance for ethnicity in MRCGP by one of us find little evidence of bias, 11 and are consistent with similar analyses of MRCP(UK) at the group level 12 and the individual examiner level. 13 Despite Esmail and Roberts' claim that "subjective bias owing to racial discrimination cannot be excluded," 7 it seems unlikely from our empirical analyses [11][12][13] that racial discrimination is an explanation for differential performance by international medical graduates in exams such as MRCGP and MRCP(UK).…”

“…Group analyses of examiner and candidate concordance for ethnicity in MRCGP by one of us find little evidence of bias, 11 and are consistent with similar analyses of MRCP(UK) at the group level 12 and the individual examiner level. 13 Despite Esmail and Roberts' claim that "subjective bias owing to racial discrimination cannot be excluded," 7 it seems unlikely from our empirical analyses [11][12][13] that racial discrimination is an explanation for differential performance by international medical graduates in exams such as MRCGP and MRCP(UK).…”

The NHS and international medical graduates

“…8,9 However, it is not news that IMGs are likely to fare worse in the current MRCGP, nor in the previous MRCGP examinations, nor in fact into the entry process to GP training. [9][10][11][12] Higher failure rates in ethnic minorities and/or IMGs have also been reported at undergraduate level, in other postgraduate exams and in other countries. [14][15][16][17][18][19] It is also known that IMGs are more likely to be complained about to the GMC and receive sanctions or warnings.…”

Selecting an internationally diverse medical workforce

“…4 Clearly, we need to explore these differences further. 5 Discrimination, which was not ruled out by Esmail and Roberts, but seems unlikely from Denney and colleagues' analysis , 6 would be wholly unacceptable. The NHS constitution makes clear that "Respect, dignity, compassion and care should be at the core of how patients and staff are treated not only because that is the right thing to do but because patient safety, experience and outcomes are all improved when staff are valued, empowered and supported."…”