Standardized Video Interviews Do Not Correlate to United States Medical Licensing Examination Step 1 and Step 2 Scores

Abstract

IntroductionIn 2017, the Standardized Video Interview (SVI) was required for applicants to emergency medicine (EM). The SVI contains six questions highlighting professionalism and interpersonal communication skills. The responses were scored (6–30). As it is a new metric, no information is available on correlation between SVI scores and other application data. This study was to determine if a correlation exists between applicants’ United States Medical Licensing Examination (USMLE) and SVI scores. We hypothesized that numeric USMLE Step 1 and Step 2 Clinical Knowledge (CK) scores would not correlate with the SVI score, but that performance on the Step 2 Clinical Skills (CS) portion may correlate with the SVI since both test communication skills.MethodsNine EM residency sites participated in the study with data exported from an Electronic Residency Application Service (ERAS®) report. All applicants with both SVI and USMLE scores were included. We studied the correlation between SVI scores and USMLE scores. Predetermined subgroup analysis was performed based on applicants’ USMLE Step 1 and Step 2 CK scores as follows: (≥ 200, 201–220, 221–240, 241–260, >260). We used linear regression, the Kruskal-Wallis test and Mann-Whitney U test for statistical analyses.Results1,325 applicants had both Step 1 and SVI scores available, with no correlation between the overall scores (p=0.58) and no correlation between the scores across all Step 1 score ranges, (p=0.29). Both Step 2 CK and SVI scores were available for 1,275 applicants, with no correlation between the overall scores (p=0.56) and no correlation across all ranges, (p=0.10). The USMLE Step 2 CS and SVI scores were available for 1,000 applicants. Four applicants failed the CS test without any correlation to the SVI score (p=0.08).ConclusionWe found no correlation between the scores on any portion of the USMLE and the SVI; therefore, the SVI provides new information to application screeners.

The Standardized Video Interview: How Does It Affect the Likelihood to Invite for a Residency Interview?

“…There are currently limited published data available on the SVI or its effects on LTI. The SVI has been shown to not correlate with USMLE scores thus providing an additional piece of data not previously available on ERAS applications . One single‐site study found no correlation between the SVI score and faculty gestalt scores on communication and professionalism …”

Computer-based test (CBT) and OSCE scores predict residency matching and National Board assessment results in Japan

“…Whether performed face-to-face or by video, an interview is inevitable for evaluating professionalism, interpersonal, and communication skills [ 22 – 24 ]. In this viewpoint, the fact that the JRMP placement and the NBE placement do not correlate with each other is very intriguing and may be related to the situation in other countries [ 25 ]. In 2007, Peskun and Shandling have reported that in Canada, residency ranking in internal medicine and family medicine is significantly correlated with second year OSCE and undergraduate grade point average [ 26 ].…”

Bias in recruitment: A focus on virtual interviews and holistic review to advance diversity

“…[24][25][26] One study revealed no correlation between USMLE scores and performance on the SVI. 27 Another study, while favoring women over men and US-MD applicants in SVI, also showed standardized evaluation ratings having bias against Black men. 28 Further an additional study found that emergency medicine interviewees were concerned about bias toward their physical appearance, ethnicity, and video production quality.…”