Experimental Philosophy and the Incentivisation Challenge: a Proposed Application of the Bayesian Truth Serum

Abstract

A key challenge in experimental social science research is the incentivisation of subjects such that they take the tasks presented to them seriously and answer honestly. If subject responses can be evaluated against an objective baseline, a standard way of incentivising participants is by rewarding them monetarily as a function of their performance. However, the subject area of experimental philosophy is such that this mode of incentivisation is not applicable as participant responses cannot easily be scored along a true-false spectrum by the experimenters. We claim that experimental philosophers’ neglect of and claims of unimportance about incentivisation mechanisms in their surveys and experiments has plausibly led to poorer data quality and worse conclusions drawn overall, potentially threatening the research programme of experimental philosophy in the long run. As a solution to this, we propose the adoption of the Bayesian Truth Serum, an incentive-compatible mechanism used in economics and marketing, designed for eliciting honest responding in subjective data designs by rewarding participant answers that are surprisingly common. We argue that the Bayesian Truth Serum (i) adequately addresses the issue of incentive compatibility in subjective data research designs and (ii) that it should be applied to the vast majority of research in experimental philosophy. Further, we (iii) provide an empirical application of the method, demonstrating its qualified impact on the distribution of answers on a number of standard experimental philosophy items and outline guidance for researchers aiming to apply this mechanism in future research by specifying the additional costs and design steps involved.

Taking a Closer Look at the Bayesian Truth Serum

“…While we cannot rule this out, we point out that the studied items were quite varied, ranging from moral permissibility judgments to views on causality and determinism. Importantly, previous work (Schoenegger, 2021) had found BTS effects for these specific items. Our results might be at odds with previous studies that did establish an effect of the BTS (e.g., Frank et al, 2017;Howie et al, 2011;Loughran et al, 2014;Weaver & Prelec, 2013), but we believe they are nevertheless interesting because they show that these effects need not generalize across samples.…”

“…Our results might be at odds with previous studies that did establish an effect of the BTS (e.g., Frank et al, 2017;Howie et al, 2011;Loughran et al, 2014;Weaver & Prelec, 2013), but we believe they are nevertheless interesting because they show that these effects need not generalize across samples. In this light, it is important to highlight that we recruited participants in the same manner as Schoenegger (2021) did. The main difference between our study and that of Schoenegger's was that we equated the duration of the No Incentive and BTS Relating to our secondary hypotheses and in line with previous work (e.g., Buhrmester et al, 2011;Crump et al, 2013;Mason & Watts, 2009;Rouse, 2015), we fail to find a significant effect of the Additional Money condition over the control.…”

“…Those in the Bayesian Truth Serum condition received an introduction to the Bayesian Truth Serum based on the original one introduced by Prelec (2004). Figure 1 contains the specific wording used in this study, which is the same as used in the stud by Schoenegger (2021).…”

“…However, while there has been significant work on the Bayesian Truth Serum and its underlying mechanisms (e.g., Frank et al, 2017;Weaver & Prelec, 2013), there remain a number of questions regarding its application that previous work has not yet addressed. In this paper, our central aim was to (a) directly replicate the most recent results by Schoenegger (2021) to ensure that the results found there robustly generalize to a new sample and that the effects of the Bayesian Truth Serum are as such also likely to replicate in other researchers' work. Further, (b) we aim to investigate if the Bayesian Truth Serum is distinct from its constituent parts, for example, by looking at whether increased monetary compensation matching the expected earnings of the participants incentivized by the Bayesian Truth Serum could explain the shift in responses, as one might wonder whether any given effect of the Bayesian Truth Serum may be primarily due to increased expected earnings, or (c) whether the addition of the prediction process itself could explain the change in response patterns by including a condition where participants are not incentivized by the Bayesian Truth Serum while still providing the same predictions.…”

“…One mechanism that allows for the introduction of an incentive-compatible design in such circumstances is the Bayesian Truth Serum (BTS; Prelec, 2004), which rewards participants based on how surprisingly common their answers are. Recently, Schoenegger (2021) applied this mechanism in the context of Likert-scale self-reports, finding that the introduction of this mechanism significantly altered response behavior. In this registered report, we further investigate this mechanism by (1) attempting to directly replicate the previous result and (2) analyzing if the Bayesian Truth Serum’s effect is distinct from the effects of its constituent parts (increase in expected earnings and addition of prediction tasks).…”