The small impact of p-hacking marginally significant results on the meta-analytic estimation of effect size

Abstract

La etiqueta p-hacking (pH) se refiere a un conjunto de prácticas oportunistas destinadas a hacer que sean significativos algunos valores p que deberían ser no significativos. Algunos han argumentado que debemos prevenir y luchar contra el pH por varias razones, especialmente debido a sus posibles efectos nocivos en la evaluación de los resultados de la investigación primaria y su síntesis meta-analítica. Nos focalizamos aquí en el efecto de un tipo específico de pH, centrado en estudios marginalmente significativos, en la estimación combinada del tamaño del efecto en el meta-análisis. Queremos saber cuánto deberíamos preocuparnos por su efecto de sesgo al evaluar los resultados de un meta-análisis. Hemos calculado el sesgo en una variedad de situaciones que parecen realistas en términos de prevalencia y de la definición operativa del pH. Los resultados muestran que en la mayoría de las situaciones analizadas el sesgo es inferior a una centésima (± 0.01), en términos de d o r. Para alcanzar un nivel de sesgo de cinco centésimas (± 0.05), tendría que haber una presencia masiva de este tipo de pH, lo que parece poco realista. Hay muchas buenas razones para luchar contra el pH, pero nuestra conclusión principal es que entre esas razones no se incluye que tenga un gran impacto en la estimación meta-analítica del tamaño del efecto. The label p-hacking (pH) refers to a set of opportunistic practices aimed at making statistically significant p values that should be non-significant. Some have argued that we should prevent and fight against pH for several reasons, especially because of its potential harmful effects on the assessment of both primary research results and their meta-analytical synthesis. We focus here on the effect of a specific type of pH, focused on marginally significant studies, on the combined estimation of effect size in meta-analysis. We want to know how much we should be concerned with its biasing effect when assessing the results of a meta-analysis. We have calculated the bias in a range of situations that seem realistic in terms of the prevalence and the operational definition of pH. The results show that in most of the situations analyzed the bias is less than one hundredth (± 0.01), in terms of d or r. To reach a level of bias of five-hundredths (± 0.05), there would have to be a massive presence of this type of pH, which seems rather unrealistic. There are many good reasons for fighting against pH, but our main conclusion is that among them is not that it has a big impact on the meta-analytical estimation of effect size.

Meta-analyzing the multiverse: A peek under the hood of selective reporting.

“…Contrary to these two simulation studies (Botella et al, 2021; Friese & Frankenbach, 2020), we found that p -hacking with actual data and using fairly generic researcher DFs could cause substantial inflation of meta-analytic average effect sizes also when the average effect appears to be null. Both Friese & Frankenbach (2020) and Botella et al (2021) run extensive simulation studies of the effect of p -hacking across many conditions and Friese & Frankenbach (2020) consider how it interacts with publication bias, something we did not do. We believe the difference in results is due to the choice in both of these simulation studies to p -hack results based on the common assumption that p -hacking leads to a peak of p -values below 0.05 (Hartgerink, 2017).…”

“…Our modeling of it in this study was relatively straightforward and we only attempted to model the outcomes of intentional selective reporting ( p -hacking). Nonetheless, our biased selection methods applied to empirical RRR data are similar to those used in the simulations of a recent compendium of p -hacking methods Stefan & Schnbrodt (2022) and are on par with other recent simulation studies of p -hacking in a meta-analysis (Botella et al, 2021; Friese & Frankenbach, 2020).…”

“…Unfortunately, there exists little evidence on which method of selection researchers use in practice. Incremental p -hacking can still lead to concerning numbers of false positive results, as discussed by both Friese & Frankenbach (2020) and Botella et al (2021), and it is important to discuss that not all types of p -hacking lead to concerning levels of effect size inflation. Nonetheless, our results show that suggesting selective reporting is not a concern for meta-analytic results is inaccurate when considering nonincremental p -hacking based on researcher DFs that we consider to be widely applicable across a range of empirical studies.…”

“…We believe the difference in results is due to the choice in both of these simulation studies to p-hack results based on the common assumption that p-hacking leads to a peak of p-values below 0.05 (Hartgerink, 2017). In the case of Friese & Frankenbach (2020), results were p-hacked to a distribution with a mode of p = .049, and Botella et al (2021), similarly p-hacked studies into the region .025 , p ≤ .05. We know from previous studies that the type of p-hacking matters; incremental methods such as optional stopping that result in a peak below p = .05 have little effect on effect size inflation, whereas methods such as outcome reporting bias have a large effect and do not result in a peak (Francis, 2012;Kirkham et al, 2010;Stefan & Schnbrodt, 2022).…”

“…Such biasing effects have been studied in simulated data for meta-analysis (e.g., Botella et al, 2021; Carter et al, 2019; Friese & Frankenbach, 2020), but simulated data from known distributions may not be representative of actual psychological data that feature unknown (distributional) complexities. Also, the effects of researcher DFs have been studied in observed data of individual studies (Botvinik-Nezer et al, 2020; Breznau et al, 2021; Huntington-Klein et al, 2021; e.g., Silberzahn et al, 2018), but not in meta-analytic context to inform how they might affect cumulative knowledge.…”