Dealing with Imperfect Elicitation Results

Abstract

The trial-and-roulette method is a popular method to extract experts' beliefs about a statistical parameter. However, most studies examining the validity of this method only use 'perfect' elicitation results. In practice, it is sometimes hard to obtain such neat elicitation results. In our project about predicting fraud and questionable research practices among Ph.D. candidates, we ran into issues with imperfect elicitation results. The goal of the current chapter is to provide an overview of the solutions we used for dealing with these imperfect results, so that others can benefit from our experience. We present information about the nature of our project, the reasons for the imperfect results and how we resolved these supported by annotated R-syntax.

The Use of Questionable Research Practices to Survive in Academia Examined With Expert Elicitation, Prior-Data Conflicts, Bayes Factors for Replication Effects, and the Bayes Truth Serum

“…More worrying, academic leaders–such as deans and heads of departments—might have a blind spot for the pressure Ph.D. candidates may experience to conduct QRPs or even fraud. Academic leaders do not always have an accurate, up-to-date perception of Ph.D. candidates’ willingness to engage in QRPs; eight leaders put all their density mass on exactly 0%, see Figure 5A in van de Schoot et al (2021b) . Some academic leaders in this study underestimated the inclination of Ph.D. candidates to conduct fraud or QRPs, although it must be said that some experts overestimated the percentage.…”

“…As shown in van de Schoot et al (2021b) , 82% (40 and 18% for scenarios 2 and 3, respectively) of the academic leaders believed the percentage of Ph.D. candidates willing to publish a paper, even if they did not trust the data because of potential data fabrication, to be precisely zero ( n = 8) or close to zero ( n = 20).…”

“…See Figure 1 for an example of such a stickered distribution and see Figure 2 for all the statistical distributions per scenario. The method we used to obtain statistical distributions based on the stickered distributions is published in van de Schoot et al (2021b) .…”

“…However, there remain many tools in the Bayesian toolbelt that are less familiar in the applied literature. In the current article, we illustrate how some less familiar tools can be applied to empirical data: A Bayesian expert-elicitation method ( O’Hagan et al, 2006 ; Anca et al, 2021 ) – also described in van de Schoot et al (2021b) , a test for prior-data conflict using the prior predictive p -value ( Box, 1980 ) and the Data Agreement Criterion (DAC) ( Veen et al, 2018 ), a Bayes truth serum to correct for socially desirable responses ( Prelec, 2004 ), and testing for replication effects via the Bayes Factor ( Bayarri and Mayoral, 2002 ; Verhagen and Wagenmakers, 2014 ). These methods are applied to the case of how Ph.D. students respond to academic publication pressure in terms of conducting questionable research practices (QRPs).…”