Reexamining How Utility and Weighting Functions Get Their Shapes: A Quasi-Adversarial Collaboration Providing a New Interpretation

Abstract

In a paper published in Management Science in 2015, Stewart, Reimers, and Harris (SRH) demonstrated that shapes of utility and probability weighting functions could be manipulated by adjusting the distributions of outcomes and probabilities on offer as predicted by the theory of decision by sampling. So marked were these effects that, at face value, they profoundly challenge standard interpretations of preference theoretic models in which such functions are supposed to reflect stable properties of individual risk preferences. Motivated by this challenge, we report an extensive replication exercise based on a series of experiments conducted as a quasi-adversarial collaboration across different labs and involving researchers from both economics and psychology. We replicate the SRH effect across multiple experiments involving changes in many design features; importantly, however, we find that the effect is also present in designs modified so that decision by sampling predicts no effect. Although those results depend on model-based inferences, an alternative analysis using a model-free comparison approach finds no evidence of patterns akin to the SRH effect. On the basis of simulation exercises, we demonstrate that the SRH effect may be a consequence of misspecification biases arising in parameter recovery exercises that fit imperfectly specified choice models to experimental data. Overall, our analysis casts the SRH effect in an entirely new light.

On the futility of estimating utility functions: Why the parameters we measure are wrong, and why they do not generalize

“…Although people were randomly assigned to choice sets, the revealed utility functions differ greatly-which means that the utility functions are, to a large part, a property of the choice set used to elicit them and not just of the risk preference of the individuals. This finding is robust, as demonstrated by a meta analysis of 13 experiments including this one (Alempaki et al 2019). Stewart et al (2015) attribute this result to the decision by sampling model (Stewart et al 2006, Stewart 2009).…”

“…We selected a random subset of 140 of these choices, subject to the constraint that each amount, probability, pair of amounts, and pair of probabilities appeared equally frequently. This condition is reported in the meta analysis in Alempaki et al (2019) as Experiment L3.e. We decided in advance to recruit 360 participants and collected the data in two simultaneous batches, one from Prolific Academic and the other from Amazon Mechanical Turk.…”

Efficient Coding and Risky Choice

“…Stewart, Reimers, and Harris (2015) and Walasek and Stewart (2015) claim to find supporting evidence for DbS by manipulating the distribution of payoffs across choice sets, similar to the manipulation in our design. However, a re-analysis of the experimental evidence finds that neither paper can be interpreted as supporting DbS (Alempaki, Canic, Mullett, Skylark, Starmer, Stewart, and Tufanod, 2019;André and de Langhe, 2020). The issue arises from the fact that behavior was analyzed on different choice sets across experimental conditions.…”

To Replicate or Not to Replicate? Exploring Reproducibility in Economics through the Lens of a Model and a Pilot Study

“…16 16 Maniadis et al (2015) discuss possible dis-incentives of initial researchers to have their work replicated, as well as recently proposed solutions. Prominent among those proposals are journals devoted to replications, special grants for funding them, large-scale preregistered replications (Camerer et al, 2016), (quasi-) adversarial collaborations (Alempaki et al, 2016) and editors enforcing direct replications as a prerequisite for publication. Maniadis et al (2015) also propose that incentivising having one's work replicated could address important shortcomings of the current system.…”