Effect Declines Are Systematic, Strong, and Ubiquitous: A Meta-Meta-Analysis of the Decline Effect in Intelligence Research

Pietschnig, Jakob; Siegel, Magdalena; Eder, Junia Sophia Nur; Gittler, Georg

doi:10.3389/fpsyg.2019.02874

Cited by 27 publications

(48 citation statements)

References 51 publications

(53 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They found that a majority of individual meta-analyses contained evidence for declining effect sizes over time but no significant, overall decline effect. Our work complements the work of Pietschnig et al (2019) by analyzing a larger and more diverse sample of meta-analyses in intelligence research and by analyzing more patterns of bias. Specifically, we analyzed 2442 primary effect sizes from 131 meta-analyses in intelligence research, published between 1984 and 2014, to estimate the average effect size, median power, and evidence of small-study effects, decline effects, US effects, and citation bias.…”

Section: Introductionmentioning

confidence: 86%

Effect Sizes, Power, and Biases in Intelligence Research: A Meta-Meta-Analysis

et al. 2020

View full text Add to dashboard Cite

In this meta-study, we analyzed 2442 effect sizes from 131 meta-analyses in intelligence research, published from 1984 to 2014, to estimate the average effect size, median power, and evidence for bias. We found that the average effect size in intelligence research was a Pearson’s correlation of 0.26, and the median sample size was 60. Furthermore, across primary studies, we found a median power of 11.9% to detect a small effect, 54.5% to detect a medium effect, and 93.9% to detect a large effect. We documented differences in average effect size and median estimated power between different types of intelligence studies (correlational studies, studies of group differences, experiments, toxicology, and behavior genetics). On average, across all meta-analyses (but not in every meta-analysis), we found evidence for small-study effects, potentially indicating publication bias and overestimated effects. We found no differences in small-study effects between different study types. We also found no convincing evidence for the decline effect, US effect, or citation bias across meta-analyses. We concluded that intelligence research does show signs of low power and publication bias, but that these problems seem less severe than in many other scientific fields.

show abstract

Section: Introductionmentioning

confidence: 86%

Effect Sizes, Power, and Biases in Intelligence Research: A Meta-Meta-Analysis

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Subsequent studies that are larger and more confirmative in nature will likely fail to find similar extreme effects, leading to a "decline" in effect size over time. 4 Previous research found evidence for decline effects in multiple scientific fields, including in intelligence research [8,26,29,86,[91][92][93][94][95].…”

Section: Tablementioning

confidence: 99%

“…We ran several additional analyses testing different time-related patterns in effect sizes, including the early-extremes effect [similar to the Proteus phenomenon, 96; also see Pietschnig et al, 2019] and several combinations of different covariates in our original analyses, but found no evidence for any systematic effects of time on effect size. We did find that across all intelligence meta-analyses, sample size seemed to increase with publication order.…”

Section: Type Of Biasmentioning

confidence: 99%

Effect Sizes, Power, and Biases in Intelligence Research: A Meta-Meta-Analysis

Nuijten¹,

Assen²,

Augusteijn³

et al. 2018

Preprint

View full text Add to dashboard Cite

In this meta-study, we analyzed 2,442 effect sizes from 131 meta-analyses in intelligence research, published from 1984 to 2014, to estimate the average effect size, median power, and evidence for bias. We found that the average effect size in intelligence research was a Pearson’s correlation of .26, and the median sample size was 60. We estimated the power of each primary study by using the corresponding meta-analytic effect as a proxy for the true effect. The median estimated power across all studies was 51.7%, with only 31.7% of the studies reaching a power of 80% or higher. We documented differences in average effect size and median estimated power between different types of in intelligence studies (correlational studies, studies of group differences, experiments, toxicology, and behavior genetics). Across all meta-analyses, we found evidence for small study effects, potentially indicating publication bias and overestimated effects. We found no differences in small study effects between different study types. We also found no convincing evidence for the decline effect, US effect, or citation bias across meta-analyses. We conclude that intelligence research does show signs of low power and publication bias, but that these problems seem less severe than in many other scientific fields.

show abstract

“…The large number of datasets that we examine in our study differ in their characteristics and potentially confounding bias may be expected to be due to various causes. Therefore, and in line with other recent meta-meta-analyses investigating publication bias (Pietschnig et al, 2019) as well as current recommendations (e.g., Coburn & Vevea, 2015;Kepes et al, 2012;van Aert et al, 2019), applying a broad array of detection methods to all sets of effect sizes was deemed to be a reasonable strategy in our analysis.…”

Section: Analyses On the Meta-analytic Levelmentioning

confidence: 56%

“…Second, we applied nine standard and more modern bias detection methods that are based on different methodological rationales in each dataset. We purposefully used a large number of heterogeneous publication bias detection methods because evidence from several meta-analytic applications (Pietschnig et al, 2019;van Aert et al, 2019) as well as simulation studies (Carter et al, 2019;Renkewitz & Keiner, 2019) have shown that different bias detection methods are not equally sensitive to different publication bias scenarios and sources.…”

Section: Analyses On the Meta-analytic Levelmentioning

confidence: 99%

Times are Changing, Bias isn’t: A Meta-Meta-Analysis on Publication Bias Detection Practices, Prevalence Rates, and Predictors in Industrial/Organizational Psychology

Siegel¹,

Eder²,

Wicherts³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Inflated or outright false effects plague Psychological Science, but advances in the identification of dissemination biases in general and publication bias in particular have helped in dealing with biased effects in the literature. However, the application of publication bias detection methods appears to be not equally prevalent across subdisciplines. It has been suggested that particularly in I/O Psychology, appropriate publication bias detection methods are underused. In this meta-meta-analysis, we present prevalence estimates, predictors, and time trends of publication bias in 128 meta-analyses that were published in the Journal of Applied Psychology (7,263 effect sizes, 3,000,000+ participants). Moreover, we reanalyzed data of 87 meta-analyses and applied nine standard and more modern publication bias detection methods. We show that (i) the bias detection method applications are underused (only 41% of meta-analyses use at least one method) but have increased in recent years, (ii) those meta-analyses that apply such methods now use more, but mostly inappropriate methods, and (iii) the prevalence of publication bias is disconcertingly high (15% to 20% show severe, 33% to 48% some bias indication) but mostly remains undetected. Although our results indicate somewhat of a trend towards higher bias awareness, they also indicate that concerns about publication bias in I/O Psychology are justified and researcher awareness about appropriate and state-of-the-art bias detection needs to be further increased. Embracing open science practices such as data sharing or study preregistration is needed to raise reproducibility and ultimately strengthen Psychological Science in general and I/O Psychology in particular.

show abstract

Effect Declines Are Systematic, Strong, and Ubiquitous: A Meta-Meta-Analysis of the Decline Effect in Intelligence Research

Cited by 27 publications

References 51 publications

Effect Sizes, Power, and Biases in Intelligence Research: A Meta-Meta-Analysis

Effect Sizes, Power, and Biases in Intelligence Research: A Meta-Meta-Analysis

Effect Sizes, Power, and Biases in Intelligence Research: A Meta-Meta-Analysis

Times are Changing, Bias isn’t: A Meta-Meta-Analysis on Publication Bias Detection Practices, Prevalence Rates, and Predictors in Industrial/Organizational Psychology

Contact Info

Product

Resources

About