Big Little Lies: A Compendium and Simulation of p-Hacking Strategies

Stefan, Angelika; Schönbrodt, Felix D.

doi:10.31234/osf.io/xy2dk

Cited by 14 publications

(18 citation statements)

References 72 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It can also be a product of human fallibility (e.g., one's conveniently forgetting about previous attempts to test the same hypothesis by alternative means) Nevertheless, p-hacking does often have important (and detrimental) consequences for the quality of research (cf. Stefan & Schönbrodt, 2022) and should therefore be examined during the hiring process. Ultimately, it is the responsibility of each individual scientist to guard against the possible influence of p-hacking on her or his own research findings, and scientists who take that responsibility more seriously should be rewarded for that.…”

Section: Identification Of Additional "Red Flags" When Evaluating Res...mentioning

confidence: 99%

Responsible Research Assessment II: A specific proposal for hiring and promotion in psychology

Gärtner¹,

Leising²,

Schönbrodt³

2022

Preprint

102

View full text Add to dashboard Cite

Traditional metric indicators of scientific productivity (e.g., journal impact factor; h-index) have been heavily criticized for being invalid and fueling a culture that focuses on the quantity, rather than the quality, of a person’s scientific output. There is now a wide-spread demand for specified alternatives to current academic evaluation practices. In a previous report, we laid out four basic principles of a more responsible research assessment in academic hiring and promotion processes (Schönbrodt et al., 2022). The present paper offers a specific proposal for how these principles may be implemented in practice: We argue in favor of broadening the range of relevant research contributions and thus propose concrete quality criteria (including ready-to-use online templates) for published research articles, data sets and research software. These criteria are supposed to be used primarily in the first phase of the assessment process. Their function is to help establish a minimum threshold of methodological rigor that candidates need to pass in order to be further considered for hiring and promotion. In contrast, the second phase of the assessment process will focus more on the actual content of candidates’ research output and necessarily use more narrative means of assessment. We hope that this proposal will help get our colleagues in the field engaged in a discussion over ways of replacing current invalid evaluation criteria with ones that relate more closely to scientific quality.

show abstract

Section: Identification Of Additional "Red Flags" When Evaluating Res...mentioning

confidence: 99%

Responsible Research Assessment II: A specific proposal for hiring and promotion in psychology

Gärtner¹,

Leising²,

Schönbrodt³

2022

Preprint

102

View full text Add to dashboard Cite

show abstract

“…Data and relevant code for this research work are stored in GitHub: and have been archived within the Zenodo repository: [120].…”

Section: Data Accessibilitymentioning

confidence: 99%

“…Data accessibility. Data and relevant code for this research work are stored in GitHub: https://github.com/astefan1/phacking_ compendium and have been archived within the Zenodo repository: https://doi.org/10.5281/zenodo.7510292 [120].…”

mentioning

confidence: 99%

Big little lies: a compendium and simulation ofp-hacking strategies

Stefan

Schönbrodt

2023

R. Soc. open sci.

Self Cite

View full text Add to dashboard Cite

In many research fields, the widespread use of questionable research practices has jeopardized the credibility of scientific results. One of the most prominent questionable research practices is p -hacking. Typically, p -hacking is defined as a compound of strategies targeted at rendering non-significant hypothesis testing results significant. However, a comprehensive overview of these p -hacking strategies is missing, and current meta-scientific research often ignores the heterogeneity of strategies. Here, we compile a list of 12 p -hacking strategies based on an extensive literature review, identify factors that control their level of severity, and demonstrate their impact on false-positive rates using simulation studies. We also use our simulation results to evaluate several approaches that have been proposed to mitigate the influence of questionable research practices. Our results show that investigating p -hacking at the level of strategies can provide a better understanding of the process of p -hacking, as well as a broader basis for developing effective countermeasures. By making our analyses available through a Shiny app and R package, we facilitate future meta-scientific research aimed at investigating the ramifications of p -hacking across multiple strategies, and we hope to start a broader discussion about different manifestations of p -hacking in practice.

show abstract

“…However, a common concern to meta-analysis is the overestimation of effect size due to publication bias, the preferential publishing of statistically significant studies (Masicampo & Lalande, 2012;Rosenthal & Gaito, 1964;Rothstein et al, 2005a;Scheel et al, 2021;Wicherts, 2017). In addition, this effect size exaggeration can be further increased by questionable research practices, that is, researchers' tendency to manipulate their data in a way that increases the effect size and the evidence for an effect (e.g., Simmons et al, 2011;Stefan & Schönbrodt, 2022). Indeed, descriptive surveys find that both problems are remarkably common.…”

Section: Robust Bayesian Meta-analysis: Model-averaging Across Comple...mentioning

confidence: 99%

Robust Bayesian meta‐analysis: Model‐averaging across complementary publication bias adjustment methods

Bartoš

Maier

Wagenmakers

et al. 2022

Research Synthesis Methods

View full text Add to dashboard Cite

Publication bias is a ubiquitous threat to the validity of meta-analysis and the accumulation of scientific evidence. In order to estimate and counteract the impact of publication bias, multiple methods have been developed; however, recent simulation studies have shown the methods' performance to depend on the true data generating process, and no method consistently outperforms the others across a wide range of conditions.Unfortunately, when different methods lead to contradicting conclusions, researchers can choose those methods that lead to a desired outcome. To avoid the condition-dependent, all-or-none choice between competing methods and conflicting results, we extend robust Bayesian meta-analysis and model-average across two prominent approaches of adjusting for publication bias: (1) selection models of p-values and (2) models adjusting for small-study effects. The resulting model ensemble weights the estimates and the evidence for the absence/presence of the effect from the competing approaches with the support they receive from the data. Applications, simulations, and comparisons to preregistered, multi-lab replications demonstrate the benefits of Bayesian model-averaging of complementary publication bias adjustment methods.

show abstract

Big Little Lies: A Compendium and Simulation of p-Hacking Strategies

Cited by 14 publications

References 72 publications

Responsible Research Assessment II: A specific proposal for hiring and promotion in psychology

Responsible Research Assessment II: A specific proposal for hiring and promotion in psychology

Big little lies: a compendium and simulation ofp-hacking strategies

Robust Bayesian meta‐analysis: Model‐averaging across complementary publication bias adjustment methods

Contact Info

Product

Resources

About