Detecting cheating in written medical examinations by statistical analysis of similarity of answers: pilot study

McManus, I. C.; Lissauer, Tom; Williams, Sutton

doi:10.1136/bmj.330.7499.1064

Cited by 22 publications

(23 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However the indices may form part of a broader set of evidence that might allow such identification, or be used to flag up behaviours which potentially are problematic for an examination. As it is, the indices certainly could provide evidence of biases which are statistically unlikely, and therefore require explanation, the logic, in many ways, being similar to that used in identifying candidates in multiple-choice examinations whose pattern of answering is ‘anomalous’, and requires explanation [11]. …”

Section: Discussionmentioning

confidence: 99%

“…If there are N examiners, then for normal scientific processes a typical Bonferroni correction would be to require that results are not significant at, say, the conventional p < .05 level but instead at the level of p < .05/ N . In many ways the problem being addressed here, of identifying individuals whose behaviour is not merely anomalous, but could possibly have implications regarding probity or professional behaviour, is similar to that of identifying candidates whose pattern of responses in a multiple-choice examination, might well be construed as cheating [11]; in that case a higher criterion of significance such as 0.001 might be felt to be desirable (and it should also be Bonferroni corrected), although in the present paper, where the analyses are on an exploratory basis, we have used only the .05 level. Of course if there is a strong a priori reason for considering the behaviour of a particular examiner (such as in a case where there is independent evidence from a separate source), then the Bonferroni correction may not be appropriate.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Investigating possible ethnicity and sex bias in clinical examiners: an analysis of data from the MRCP(UK) PACES and nPACES examinations

2013

Self Cite

View full text Add to dashboard Cite

BackgroundBias of clinical examiners against some types of candidate, based on characteristics such as sex or ethnicity, would represent a threat to the validity of an examination, since sex or ethnicity are ‘construct-irrelevant’ characteristics. In this paper we report a novel method for assessing sex and ethnic bias in over 2000 examiners who had taken part in the PACES and nPACES (new PACES) examinations of the MRCP(UK).MethodPACES and nPACES are clinical skills examinations that have two examiners at each station who mark candidates independently. Differences between examiners cannot be due to differences in performance of a candidate because that is the same for the two examiners, and hence may result from bias or unreliability on the part of the examiners. By comparing each examiner against a ‘basket’ of all of their co-examiners, it is possible to identify examiners whose behaviour is anomalous. The method assessed hawkishness-doveishness, sex bias, ethnic bias and, as a control condition to assess the statistical method, ‘even-number bias’ (i.e. treating candidates with odd and even exam numbers differently). Significance levels were Bonferroni corrected because of the large number of examiners being considered.ResultsThe results of 26 diets of PACES and six diets of nPACES were examined statistically to assess the extent of hawkishness, as well as sex bias and ethnicity bias in individual examiners. The control (odd-number) condition suggested that about 5% of examiners were significant at an (uncorrected) 5% level, and that the method therefore worked as expected. As in a previous study (BMC Medical Education, 2006, 6:42), some examiners were hawkish or doveish relative to their peers. No examiners showed significant sex bias, and only a single examiner showed evidence consistent with ethnic bias. A re-analysis of the data considering only one examiner per station, as would be the case for many clinical examinations, showed that analysis with a single examiner runs a serious risk of false positive identifications probably due to differences in case-mix and content-specificity.ConclusionsIn examinations where there are two independent examiners at a station, our method can assess the extent of bias against candidates with particular characteristics. The method would be far less sensitive in examinations with only a single examiner per station as examiner variance would be confounded with candidate performance variance. The method however works well when there is more than one examiner at a station and in the case of the current MRCP(UK) clinical examination, nPACES, found possible sex bias in no examiners and possible ethnic bias in only one.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Investigating possible ethnicity and sex bias in clinical examiners: an analysis of data from the MRCP(UK) PACES and nPACES examinations

2013

Self Cite

View full text Add to dashboard Cite

show abstract

“…The program was initially used in 2003 in the MRCPCH examination and reported in the BMJ in 2005. 6 Acinonyx considers the similarity of the answers of all possible pairs of candidates taking an examination, irrespective of whether the candidates are sitting in the same centre or different centres and identifies anomalous pairs of candidates, ie those whose answers are significantly more alike than would be expected by chance. The program provides evidence that requires further investigation, such as scrutiny of seating plans, question booklets for notes and changed answers, information from invigilators and interviews with candidates, before actual cheating can be proved.…”

Section: Candidates Have a Duty To Report (To An Invigilator Or Mrcp(uk)mentioning

confidence: 99%

Dishonesty in the MRCP(UK) Part 1 and Part 2 written examinations: prevention, detection and possible remediation

2011

View full text Add to dashboard Cite

“…EDITOR—McManus et al suggest a promising initial step towards identifying occurrences of potential academic dishonesty 1. A computer program, however, should not be seen as the final decision or a solution to the problem, although it is an effective tool.…”

mentioning

confidence: 99%