Bias, Fairness, and Validity in Graduate Admissions: A Psychometric Perspective

Abstract

As many schools and departments are considering the removal of the Graduate Record Examination (GRE) from their graduate admission processes to enhance equity and diversity in higher education, controversies arise. From a psychometric perspective, we see a critical need for clarifying the meanings of measurement bias and fairness, in order to create common ground for constructive discussions within the field of psychology, higher education, and beyond. We critically evaluate six major sources of information that are widely used in graduate admission assessment: grade point average, personal statements, resumes/CVs, letters of recommendation, interviews, and GRE. We review empirical research evidence available to date on the validity, bias, and fairness issues associated with each of these admission measures, and identify potential issues that have been overlooked in the literature. We conclude by suggesting several directions for practical steps to improve the current admissions decisions, as well as highlighting areas in which future research would be beneficial.

Revising evaluation metrics for graduate admissions and faculty advancement to dismantle privilege

“…In raising the potential for flexibility in holistic models, it is important to nonetheless consider the value of standardization, at least within specific models. For instance, with regards to graduate admissions, it has been suggested that the use of structured and standardized materials (personal statements, interviews, letters of recommendation) may be one way to guard against the kinds of cognitive biases that might contribute to racial disparities 6 . We would emphasize that standardization of any element of an application package along these lines need not necessitate that all programs adopt the same set of criteria.…”

“…Empirical evidence suggests that undergraduate GPA, while correlated with graduate comprehensive exam scores and faculty ratings of graduate student performance, is not a very strong predictor of degree attainment or time to completion 5 . Personal statements and letters of recommendation are likewise limited in their ability to predict future academic performance, and they are vulnerable to a number of sociocognitive and rater biases 6 . Most surprisingly, previous research experience also appears to be unrelated to other predictors used in graduate admissions, as well as academic performance in graduate school, although this warrants further investigation, as only a small number of studies have been conducted on this topic 7 .…”

Review of Racially Equitable Admissions Practices in STEM Doctoral Programs

“…Reviewing each part of an application separately (for example, all applicants' research experience, and then all applicants' letters of recommendation) can also reduce cognitive bias. Standardizing the review process is extremely important because bias in the evaluation of an application is not limited to the GRE, but can also appear in the undergraduate GPA, letters of recommendation, and in access to research opportunities [31]. Public and private undergraduate degree-granting institutions give out different percentages of A's and B's, as do institutions in different geographic regions [32].…”

“…For example, even if the language in the letter is overall very positive, readers may interpret "X is brilliant" to mean "X is brilliant (for X's group)" if X has an underrepresented identity in that field [35,36]. Finally, evaluating research experience and evidence of research project completion ought to be modulated by consideration of the undergraduate institution and other work experience [22,31]. The applicant may not have had access to opportunities for research at their previous institution.…”

Commentary: Reviving the Messenger: A Response to Koljatic et al. (2021)

“…Unlike test scores, much less validity evidence exists on these alternate measures. The research that does exist suggests that, as compared to test scores, these subjective measures are not as predictive of future success, are more prone to cognitive biases, and thus are likely to exhibit larger group differences by socioeconomic status (SES) given the inequities in resources discussed above (e.g., Alvero et al., 2021; Murphy, Klieger, Borneman, & Kuncel, 2009; Woo, LeBreton, Keith, & Tay, 2020). This may explain why research on test optional policies has found minimal to no effect on diversity outcomes (e.g., Belasco, Rosinger, & Hearn, 2015; Bennett, 2021).…”