The Autonomy-validity Dilemma in Mechanical Prediction Procedures: The Quest for a Compromise

Abstract

A robust finding in psychological research is that combining information with a mechanical rule results in more valid predictions than combining information holistically in the mind. Nevertheless, information is typically combined holistically in practice, resulting in suboptimal predictions and decisions. Earlier research showed that decision makers are more likely to use mechanical prediction procedures when they retain autonomy in the decision-making process. However, it remains largely unknown how different autonomy-enhancing features affect predictive validity. Therefore, in two pre-registered studies (total N = 342), we investigated if and how prediction procedures can be designed such that they satisfy decision-makers’ autonomy needs and acceptance without reducing predictive validity. Based on archival application data from a university admission procedure, participants predicted applicants’ first-year GPA and chance of dropout. The results of Bayesian analyses showed that participants preferred prediction procedures in which they retained autonomy by choosing consistent predictor weights of a mechanical rule or by holistically adjusting the predictions of an optimal regression model. In general, these prediction procedures resulted in slightly higher predictive validity compared to fully holistic prediction. Providing participants with predictor validity information slightly increased predictive validity when participants could choose predictor weights, but not when making holistic predictions or adjusting optimal model predictions. Giving decision makers a role in designing mechanical rules through choosing weights based on explicit predictive validity information could help promote the implementation and validity of mechanical prediction in practice.

Education increases decision-rule use: An investigation of education and incentives to improve decision making.

“…Furthermore, the decision maker virtually always has more information than is included in the decision rule (Grove & Meehl, 1996, p. 297). Moreover, a rating of the personal statement would have received a weight of zero in the decision rule because another study based on these data shows that it does not provide any incremental validity over and above high school GPA and the admission test score (Neumann et al, 2021).…”