Initial investigation into computer scoring of candidate essays for personnel selection.

Abstract

[Correction Notice: An Erratum for this article was reported in Vol 101(7) of Journal of Applied Psychology (see record 2016-32115-001). In the article the affiliations for Emily D. Campion and Matthew H. Reider were originally incorrect. All versions of this article have been corrected.] Emerging advancements including the exponentially growing availability of computer-collected data and increasingly sophisticated statistical software have led to a "Big Data Movement" wherein organizations have begun attempting to use large-scale data analysis to improve their effectiveness. Yet, little is known regarding how organizations can leverage these advancements to develop more effective personnel selection procedures, especially when the data are unstructured (text-based). Drawing on literature on natural language processing, we critically examine the possibility of leveraging advances in text mining and predictive modeling computer software programs as a surrogate for human raters in a selection context. We explain how to "train" a computer program to emulate a human rater when scoring accomplishment records. We then examine the reliability of the computer's scores, provide preliminary evidence of their construct validity, demonstrate that this practice does not produce scores that disadvantage minority groups, illustrate the positive financial impact of adopting this practice in an organization (N ∼ 46,000 candidates), and discuss implementation issues. Finally, we discuss the potential implications of using computer scoring to address the adverse impact-validity dilemma. We suggest that it may provide a cost-effective means of using predictors that have comparable validity but have previously been too expensive for large-scale screening. (PsycINFO Database Record

Quantifying with words: An investigation of the validity of narrative‐derived performance scores

“…Finally, like Campion et al. (), this study highlights the utility of leveraging text mining within organizational settings. Within this domain, there is room to investigate the many methods and approaches for scoring text data.…”

“…Third, this paper looks at when narratives might be most informative and in what location of the traditional rating scale narrative scores would be most variable and explain the most unique variance. Finally, like Campion, Campion, Campion, and Reider (2016), this paper demonstrates ways to leverage recent enhancements in computer technology to better understand phenomena in the organizational sciences. Altogether, this paper investigates how comment data can be conceptualized and easily extracted to improve performance measurement within organizations.…”

Highly automated job interviews: Acceptance under the influence of stakes

“…Langer et al, ) and because highly automated solutions might be more useful to screen a large applicant pool (cf. Campion et al, ), which is more likely for entry‐level positions compared to jobs in higher management.…”

“…Automatically implementing actions means to actually implement the action decisions. At lower levels of automation, this could mean to automatically rank interviewees regarding their performance and presenting this ranking to hiring managers (Campion, Campion, Campion, & Reider, ). At higher levels of automation this might mean to automatically choose appropriate follow‐up questions from a set of questions for a given participant or to automatically screen out certain applicants (cf.…”

When your resume is (not) turning you down: Modelling ethnic bias in resume screening

“…Although professionals still prefer holistic information processing (Kuncel, Klieger, & Ones, 2014), Kuncel, Klieger, Connelly, and Ones (2013) showed that mechanical data combination methods resulted in more than 50% improvement in the prediction of work and academic criteria when compared with more holistic, intuitive methods. Other researchers further showed that algorithms can rate applicants' accomplishment narratives as reliably as human raters (Campion, Campion, Campion, & Reider, 2016), can predict applicants' personality traits and social/communication skills reasonably well from nonverbal cues extracted from video resumes (Nguyen & Gatica-Perez, 2016) or from Facebook likes (Youyou, Kosinski, & Stillwell, 2015), and can even predict which candidates would most likely become involved in shooting or be accused of abuse as police officers (Chalfin et al, 2016).…”