Times are Changing, Bias isn’t: A Meta-Meta-Analysis on Publication Bias Detection Practices, Prevalence Rates, and Predictors in Industrial/Organizational Psychology

Abstract

Inflated or outright false effects plague Psychological Science, but advances in the identification of dissemination biases in general and publication bias in particular have helped in dealing with biased effects in the literature. However, the application of publication bias detection methods appears to be not equally prevalent across subdisciplines. It has been suggested that particularly in I/O Psychology, appropriate publication bias detection methods are underused. In this meta-meta-analysis, we present prevalence estimates, predictors, and time trends of publication bias in 128 meta-analyses that were published in the Journal of Applied Psychology (7,263 effect sizes, 3,000,000+ participants). Moreover, we reanalyzed data of 87 meta-analyses and applied nine standard and more modern publication bias detection methods. We show that (i) the bias detection method applications are underused (only 41% of meta-analyses use at least one method) but have increased in recent years, (ii) those meta-analyses that apply such methods now use more, but mostly inappropriate methods, and (iii) the prevalence of publication bias is disconcertingly high (15% to 20% show severe, 33% to 48% some bias indication) but mostly remains undetected. Although our results indicate somewhat of a trend towards higher bias awareness, they also indicate that concerns about publication bias in I/O Psychology are justified and researcher awareness about appropriate and state-of-the-art bias detection needs to be further increased. Embracing open science practices such as data sharing or study preregistration is needed to raise reproducibility and ultimately strengthen Psychological Science in general and I/O Psychology in particular.

Times are changing, bias isn’t: A meta-meta-analysis on publication bias detection practices, prevalence rates, and predictors in industrial/organizational psychology.

“…For example, more than 90% of the presently investigated meta-analyses used a Hunter and Schmidt approach, a method that focuses on alleviating potentially effect-deflating effects due to statistical artifacts such as unreliability or range variation (Hunter & Schmidt, 2004). In many other Psychology subfields outside of I/O Psychology, the Hedges and Olkin (1985) approach—typically focusing more on potential effect-inflation—seems to be more popular (e.g., used in 76%–98% of meta-analyses in Personality and Social Psychology as well as multidisciplinary Psychology; Siegel et al, 2019). In addition, most available software packages to date that are used to conduct Hunter and Schmidt-typed meta-analyses do not include routines to address publication bias (excepting cumulative meta-analysis, Schmidt & Hunter, 2015).…”