Reliability of residents’ assessments of their postgraduate medical education learning environment: an observational study

Abstract

Background. Even in anonymous evaluations of a postgraduate medical education (PGME) program, residents may be reluctant to provide an honest evaluation of their PGME program, because they fear embarrassment or repercussions from their supervisors if their anonymity as a respondent is endangered. This study was set up to test the hypothesis that current residents in a PGME program provide more positive evaluations of their PGME program than residents having completed it. We therefore compared PGME learning environment evaluations of current residents in the program to leaving residents having completed it. Methods. This observational study used data gathered routinely in the quality cycle of PGME programs at two Dutch teaching hospitals to test our hypothesis. At both hospitals, all current PGME residents are requested to complete the Scan of Postgraduate Education Environment Domains (SPEED) annually. Residents leaving the hospital after completion of the PGME program are also asked to complete the SPEED after an exit interview with the hospital’s independent residency coordinator. All SPEED evaluations are collected and analysed anonymously. We compared the residents’ grades (on a continuous scale ranging from 0 (poor) to 10 (excellent)) on the three SPEED domains (content, atmosphere, and organization of the program) and their mean (overall department grade) between current and leaving residents. Results. Mean (SD) overall SPEED department grades were 8.00 (0.52) for 287 current residents in 39 PGME programs and 8.07 (0.48) for 170 leaving residents in 39 programs. Neither the overall SPEED department grades (t test, p=0.53, 95% CI for difference -0.16 to 0.31) nor the department SPEED domain grades (MANOVA, F(3, 62)=0.79, p=0.51) were significantly different between current and leaving residents. Conclusions. Residents leaving the program did not provide more critical evaluations of their PGME learning environment than current residents in the program. This suggests that current residents’ evaluations of their postgraduate learning environment were not affected by social desirability bias or fear of repercussions from faculty.