Gender bias in resident evaluations: Natural language processing and competency evaluation

Abstract

Background Research shows that female trainees experience evaluation penalties for gender non‐conforming behaviour during medical training. Studies of medical education evaluations and performance scores do reflect a gender bias, though studies are of varying methodology and results have not been consistent. Objective We sought to examine the differences in word use, competency themes and length within written evaluations of internal medicine residents at scale, considering the impact of both faculty and resident gender. We hypothesised that female internal medicine residents receive more negative feedback, and different thematic feedback than male residents. Methods This study utilised a corpus of 3864 individual responses to positive and negative questions over the course of six years (2012‐2018) within Yale University School of Medicine's internal medicine residency. Researchers developed a sentiment model to assess the valence of evaluation responses. We then used natural language processing (NLP) to evaluate whether female versus male residents received more positive or negative feedback and if that feedback focussed on different Accreditation Council for Graduate Medical Education (ACGME) core competencies based on their gender. Evaluator‐evaluatee gender dyad was analysed to see how it impacted quantity and quality of feedback. Results We found that female and male residents did not have substantively different numbers of positive or negative comments. While certain competencies were discussed more than others, gender did not seem to influence which competencies were discussed. Neither gender trainee received more written feedback, though female evaluators tended to write longer evaluations. Conclusions We conclude that when examined at scale, quantitative gender differences are not as prevalent as has been seen in qualitative work. We suggest that further investigation of linguistic phenomena (such as context) is warranted to reconcile this finding with prior work.

Predictors of faculty narrative evaluation quality in medical school clerkships

“…Our finding that faculty gender was statistically and practically associated with narrative evaluation quality builds upon prior studies showing that women faculty provide a greater amount 25,26,40 and higher quality of feedback 24 to trainees than men faculty. In terms of why women faculty members' narratives are higher quality, prior work has speculated that it may result from enduring gender disparities in career advancement.…”

“…The results of this study add to the broader literature of narrative evaluation quality and its supporting structural factors and processes, which to date has been relatively limited to graduate medical education settings. 15,23,25,26,40 Our finding linking narrative quality with timely completion intuitively suggests that composing narrative evaluations more temporally proximate to trainee interactions may yield higher quality narratives, though the effect size was modest relative to other study findings and prior work by Dudek and Colleagues 44 who categorised an 8% difference in the Completed Clinical Evaluation Report Rating (CCERR) as moderate to large. An equivalent percentage for the NEQI would equate to a 1-point difference.…”

“…In terms of why women faculty members' narratives are higher quality, prior work has speculated that it may result from enduring gender disparities in career advancement. 40 Put another way, the tendency for women faculty to compose high-quality narratives may be a compensatory process stemming from persistent structural and psychological barriers that inform expectations, preferences, commitments and routine interactions in inherently gendered contexts. 53 Thus, higher quality narratives in women may result from an internalised need to work harder to achieve gender parity in career advancement.…”

“…When narrative quality was examined at the student-level, we did not observe differential effects by student gender nor an interaction between faculty and student gender. With respect to the latter, work has shown mixed results regarding the effects of trainee-supervisor gender (dis)congruent pairings on narrative feedback 24,26,40 ; however, these studies are limited to resident populations. The lack of an association between narrative quality and student gender is promising but does not obviate the potential for bias in faculty narratives.…”

Assessing Group-level Gender Bias in Professional Evaluations: The Case of Medical Student End-of-Shift Feedback

“…To investigate this question more thoroughly, we use a new dataset of written assessments on medical students' work based on individual shift performance before their residencies. Most previous work from the medical community has used relatively simple linguistic methods such as the Linguistic Inquiry and Word Count dictionary (LIWC) (Pennebaker et al, 2015;Madera et al, 2009;S et al, 2017;Schmader et al, 2007), but using pretrained language models may allow us to investigate bias in a more fine-grained manner (Andrews et al, 2021;Sarraf et al, 2021). Additionally, existing work on medical bias within the NLP community mainly focuses on patients, rather than physicians themselves (van Aken et al, 2021).…”