The role of the interview in residency selection: A mixed‐methods study

Abstract

Context: Residency programmes invest considerable time and resources in candidate interviews as a result of their perceived ability to reveal important social traits. However, studies examining the ability of interviews to predict resident performance have shown mixed findings, and the role of the interview in candidate evaluation remains unclear. This mixed-methods study, conducted in an anaesthesiology residency programme at a large academic medical centre, examined how interviews contributed to candidate assessment and whether the addition of behavioural questions to interviews altered their role in the evaluation process. Methods: During the 2018-2019 residency selection season in the Department of Anesthesiology and Critical Care at the University of Pennsylvania, independent ratings for each interviewee were collected from faculty interviewers. Consensus ratings subsequently established by committee were also collected. Committee meetings were audiorecorded and transcribed for qualitative analysis. Behavioural questions were integrated into half of interview days. Ratings of candidates interviewed on behavioural question days were compared statistically with those of candidates interviewed on non-behavioural question days. Results: Qualitative analysis showed that interviewers heavily emphasised candidates' application files in evaluating the interviews. Interviewers focused on candidates' academic records and favoured candidates whose interview behaviours were consistent with their applications and whose applications demonstrated similarities to interviewers' traits. The addition of behavioural questions demonstrated little ability to alter these dynamics. Quantitatively, there were no significant differences in candidate rating outcomes between behavioural and non-behavioural interviewing days, whereas a higher medical school rating and higher score on the United States Medical Licensing Examination Step 1 were associated with a more favourable consensus rating. Conclusions: Residency candidates' application files predisposed interviewers' experience and evaluation of interviews, preventing the interviews from providing discrete assessments of interpersonal qualities, even when behavioural questions were included. In the continued effort to perform well-rounded assessments of residency

Planning virtual residency interviews as a result of COVID-19: insight from residency applicants and physicians conducting interviews

“…Thus, we can ensure that interviews benefit programmes, students and eventually patients. 8 In consideration of the findings within this study, some recommendations for residency programmes as they prepare for virtual interviews in the future include: ► Provide individual interviews over group interviews when possible. ► Increase opportunities for applicants to fully understand the programme such as development of marketing materials on the programme and city of programme that are easily Physicians (n=51) and medical residency applicants (n=104) were asked to respond to statements related to residency interviews moving to virtual and other facets of the residency interview process from strongly disagree (1) to strongly agree (5).…”

Selecting trainees: Too much focus on predictive metrics, not enough on holistic review

“…But from the perspective of construct validity, the picture is bleaker still. There is reason to suspect that many residency interviews reinforce pre‐existing differences between candidates instead of providing a discrete means of assessment 6,7 . USMLE Step 1 is a licencing examination whose ‘off‐label’ use to measure the promise of medical students has been justifiably condemned, 8 with the consequence that from 2022 onward, it will be scored pass/fail.…”

“…And there have been so few empirical studies of how residency selection committees make decisions that we have little basis on which to assess the meaningfulness of the rankings they give candidates. The studies that do exist suggest that committees are vulnerable to numerous biases when they determine rankings 6,9 …”

Designing for value in medical residency selection

“…In this issue of Medical Education , Gordon et al 1 demonstrate how interviewer ratings can be influenced by prior knowledge about the applicant such as awareness of their medical licence examination scores, CV, deans’ letters, recommendation letters, personal statements, and medical school transcripts. This long list of selection tools used within one residency selection process is reflective of the growing number of desirable attributes identified in selection literature.…”

“…The value of selection tools may be maximised through constructive alignment across the battery of tools implemented in a selection process 6 . Often, selection tools are combined in such a way that duplicates or hinders the intended purpose of each tool ‐ the study by Gordon et al 1 provides examples of both of these. Regarding the former, the selection process described included written character statements (personal statement, deans’ letters, etc) and repeated the evaluation of applicant character through the interview process.…”