The selection and training of examiners for clinical examinations

Abstract

The inconsistency of the marking in clinical examinations is a well documented problem. This project identified some of the factors responsible for this inconsistency. A standardized rating situation was devised. Five students were videotaped as they performed part of a physical examination on simulated patients. Eighteen experienced medical and surgical examiners rated their performances using an objective checklist type of rating form. No differences were evident between physicians and surgeons. The group of examiners was divided into three subgroups, one receiving no training, one limited training and one more extensive training. Examiners re-rated the same students 2 months after the first rating. Inter-rater reliability was satisfactory for the first ratings and training produced no significant improvement. A substantial improvement was achieved by identifying the most inconsistent raters and removing them from the analysis. Training was shown to be unnecessary for consistent examiners and ineffective for examiners who were less consistent. On the basis of these results, only consistent examiners were selected to take part in the interactive component of the objective structured final year examinations. The ratings in these examinations achieved high levels of inter-rater reliability. It was concluded that the combination of an objective check-list rating form, a controlled test situation and the selection of inherently consistent examiners could solve the problem of inconsistent marking in clinical examinations.

SPECIAL ARTICLE: Cognitive, Social and Environmental Sources of Bias in Clinical Performance Ratings

“…More research is needed to determine whether adding other types of more extensive training will yield higher quality clinical performance ratings by physicians. The research by Newble and associates 85 suggests that eliminating raters who are extreme outliers (positive and negative) is a more cost-effective means of increasing the validity and reproducibility of clinical performance ratings than is training. It would be nice to have other studies investigate this practice to increase confidence in the reproducibility of the finding.…”

“…They found that physicians who received training were no more accurate than other raters who participated in the study but without training. Newble et al 85 compared the interrater agreement for groups of physicians with no training, moderate training, and more extensive training. They reported that training did not improve the low rate of interrater agreement.…”

“…There was no practice and feedback component. The Newble et al 85 study involved a ½ hr training session with no feedback for the moderate training group and a 2½ hr training session including practice and feedback for the extensively trained group. The amount and type of training in the Newble et al 85 study is more in line with what would be expected to result in a positive training outcome, but it may take even more training to yield the desired improvement in reliability.…”

Factors associated with success in the oral part of the European Diploma in Intensive Care

“…There is evidence in the literature showing that examiners' training helps to standardize the exam. [8][9][10] In EDIC part 2, each candidate faces 9 different examiners who are carefully selected (ESICM-SOP) and have passed a standardized training, 11 which includes a mandatory train-the-examiner course and pre-exam workshop. In addition, before becoming an active examiner, it is mandatory to observe at least one session of the exam.…”

Inter‐rater and test–retest reliability of the Taiwanese rehabilitation functional scale

“…A proper rater training (Newble et al, 1980;Streiner and Norman, 1989;Van Deusen and Brunt, 1997) is also necessary to ensure the stability of the rater's rating. We adapted a training protocol for the occupational therapists participating in the study of TaRFS.…”