Analysis of the Evaluative Components on the Standard Letter of Recommendation (SLOR) in Emergency Medicine

Abstract

IntroductionThe standard letter of recommendation in emergency medicine (SLOR) was developed to standardize the evaluation of applicants, improve inter-rater reliability, and discourage grade inflation. The primary objective of this study was to describe the distribution of categorical variables on the SLOR in order to characterize scoring tendencies of writers.MethodsWe performed a retrospective review of all SLORs written on behalf of applicants to the three Emergency Medicine residency programs in the University of Arizona Health Network (i.e. the University Campus program, the South Campus program and the Emergency Medicine/Pediatrics combined program) in 2012. All “Qualifications for Emergency Medicine” and “Global Assessment” variables were analyzed.Results1457 SLORs were reviewed, representing 26.7% of the total number of Electronic Residency Application Service applicants for the academic year. Letter writers were most likely to use the highest/most desirable category on “Qualifications for EM” variables (50.7%) and to use the second highest category on “Global Assessments” (43.8%). For 4-point scale variables, 91% of all responses were in one of the top two ratings. For 3-point scale variables, 94.6% were in one of the top two ratings. Overall, the lowest/least desirable ratings were used less than 2% of the time.ConclusionsSLOR letter writers do not use the full spectrum of categories for each variable proportionately. Despite the attempt to discourage grade inflation, nearly all variable responses on the SLOR are in the top two categories. Writers use the lowest categories less than 2% of the time. Program Directors should consider tendencies of SLOR writers when reviewing SLORs of potential applicants to their programs.

Use of standardized letters of recommendation for otolaryngology head and neck surgery residency and the impact of gender

“…A small percentage of OTO LORs were written by community OTOs (4.2%, n 5 35). Of the remaining 123 letters written by non-OTOs, 36.6% (n 5 45) were composed by general surgeons, 24.4% (n 5 30) by surgeons in other subspecialties, 13.0% (n 5 16) by internists, 16.3% (n 5 20) by other nonsurgical physicians, 7.3% (n 5 9) by researchers (with Ph.D. degrees), and 2.4% (n 5 3) by writers whose position was not specified. Letter writers were located throughout the United States, including Puerto Rico.…”

“…The SLOR has also been purported to reduce the grandiosity often seen in narrative letters; nevertheless, subjective and objective analyses have indicated that letter writers continue to consistently select the highest ranks when reviewing student performance. The SLOR thus fails to achieve one of its goals and yields a seemingly homogenous pool of applicants . Multiple studies have shown that when choosing responses in a Likert‐type scale, as is utilized in the SLOR (Fig.…”

“…The SLOR thus fails to achieve one of its goals and yields a seemingly homogenous pool of applicants. [16][17][18] Multiple studies have shown that when choosing responses in a Likert-type scale, as is utilized in the SLOR (Fig. 1), individuals are biased to select responses on the left side.…”

Trends in NRMP Data from 2007-2014 for U.S. Seniors Matching into Emergency Medicine

“…Test scores and memberships in an honorary society should not be taken as evidence of an entirely unchanging applicant pool. However, in light of previously reported difficulties that EM faculty members have in accurately assessing applicants for letters of evaluation and in predicting position on rank lists, it is important that the relative meaning of these scores and designations be understood 14, 15…”

Dear Program Director: Deciphering Letters of Recommendation

“…[1][2][3][4][5][6][7] Concerns regarding the LOR as an accurate assessment of applicants were raised as long as 35 years ago, 8 with studies showing grade inflation in LORs. [9][10][11] One study reported that less than 2% of candidates were rated using the lowest categories, 9 while another demonstrated that 40% of candidates were rated in the top 10% on a global assessment. 10 When faculty members write LORs, they walk the fine line between writing an honest letter supporting the applicant and diminishing the credibility of these letters by overselling an average candidate.…”