Using narratives and numbers in performance prediction: Attitudes, confidence, and validity

Abstract

In a preregistered prediction-task experiment, we investigated the effect of narrative versus quantified information on decision-maker perceptions, confidence, predictor weighting, and predictive accuracy when making performance predictions. We also investigated the effect of who quantifies information (the decision maker or someone else).As expected, we found higher perceived informativeness and use intentions for narrative than quantified information. Information presented narratively was also weighted somewhat more heavily than quantified information. Using quantitative information quantified by decision makers themselves yielded higher perceived autonomy and use intentions than quantitative information quantified by someone else. However, no differences in prediction confidence were found and self-and other-produced quantifications received identical weight. Moreover, unexpectedly, differences in weighting did not translate to differences in predictive accuracy.

“Adding an Egg” in Algorithmic Decision Making: Improving Stakeholder and User Perceptions, and Predictive Validity by Enhancing Autonomy

“…Intriguingly, our results also showed that strictly using the single best predictor in a top-down manner would have resulted in more valid predictions (r = .31) than holistic prediction (r = .16). This demonstrates again that 'less can be more'; invalid information can distract decision makers from valid information and thereby reduce predictive validity of the selection system compared to using the single best predictor alone, also known as the 'dilution effect' (Dana et al, 2013;Niessen et al, 2021). Relatedly, we also showed that conscientiousness and the unstructured interview had only slight incremental validity over and above cognitive ability when this information was combined through our meta-analytic, prescribed algorithm, but not when combined holistically or through a self-designed algorithm.…”

“…Rather than a limitation, a consequence of using the applicant data from Kausel et al (2016) was that we presented decision makers with quantitative test scores and a quantified rating from an interview conducted by someone else. It is plausible that we would have obtained different results if decision makers had conducted interviews with applicants themselves, as they may have been more likely to use and overweight such qualitative and 'rich' information than a quantified interview rating from someone else (Kuncel, 2018;Niessen et al, 2021;Slaughter & Kausel, 2013). Therefore, future research could use different types of applicant data, and vary the extent to which decision makers are involved in the data collection and/or presented with qualitative or quantitative information.…”

Personnel selection as judgment and decision science: An introduction

“…Furthermore, they investigate the effect of narrative or quantified information on predictive validity. In addition, the contributions by Highhouse et al (2022) and Niessen et al (2022) also illustrate a longstanding debate on the definition of "dilution. "…”

“…In this special issue, we present four contributions to judgment and decision‐making in personnel selection. Central topics are: How assessment information is used to make predictions, how that influences judgments and decisions (Highhouse et al, 2022; Niessen et al, 2022; Yu & Kuncel, 2022), and how predictors are chosen and evaluated (Childers et al, 2022).…”

“…In “Using narratives or numbers in performance prediction: Attitudes, confidence and validity,” Niessen et al (2022) investigate whether when making performance predictions, the information presented in narrative form and obtained by the decision‐makers themselves instead of others is perceived more favorably, yield higher confidence, and is weighted more heavily than numeric information. Furthermore, they investigate the effect of narrative or quantified information on predictive validity.…”