Effect Sizes, Power, and Biases in Intelligence Research: A Meta-Meta-Analysis

Abstract

In this meta-study, we analyzed 2,442 effect sizes from 131 meta-analyses in intelligence research, published from 1984 to 2014, to estimate the average effect size, median power, and evidence for bias. We found that the average effect size in intelligence research was a Pearson’s correlation of .26, and the median sample size was 60. We estimated the power of each primary study by using the corresponding meta-analytic effect as a proxy for the true effect. The median estimated power across all studies was 51.7%, with only 31.7% of the studies reaching a power of 80% or higher. We documented differences in average effect size and median estimated power between different types of in intelligence studies (correlational studies, studies of group differences, experiments, toxicology, and behavior genetics). Across all meta-analyses, we found evidence for small study effects, potentially indicating publication bias and overestimated effects. We found no differences in small study effects between different study types. We also found no convincing evidence for the decline effect, US effect, or citation bias across meta-analyses. We conclude that intelligence research does show signs of low power and publication bias, but that these problems seem less severe than in many other scientific fields.

Distinguishing specific from general effects in cognition research.

“…Next, we discuss the feasibility of using a latent variable model instead of traditional ANOVA methods, when the sample sizes per group are small ( N = 50 to N = 100), and the expected effect size is small to moderate (Cohen’s d = 0.40). These estimates coincide with the median sample size ( N = 60) and median weighted effect size ( r = 0.26) found in approximately 2400 effect sizes in 131 meta-analyses in intelligence research (Nuijten, van Assen, Augusteijn, Crompvoets, & Wicherts, 2019).…”

How Powerful is the Evidence in Criminology? On Whether We Should Fear a Coming Crisis of Confidence

“…The figure has a distinct U shape, with a cluster of very low-powered studies on the left and a cluster of high-powered studies on the right. In that respect, the distribution looks very similar to statistical power estimates observed in other behavioral science disciplines (for a similar pattern from neuroscience, see Button et al, 2013;Nuijten et al, 2018).…”

“…Adopting the strategies that have been applied in psychology (Bakker et al, 2012;Nuijten et al, 2018) and neuroscience (Button et al, 2013), we gathered average effect size and average sample size (n) estimates from criminology meta-analyses. There are several benefits to relying on evidence from meta-analyses.…”

“statcheck”: Automatically detect statistical reporting inconsistencies to increase reproducibility ofmeta‐analyses

“…After updating statcheck, we used it to analyze 402 meta-analyses published in the social sciences. Our sample derived from three locations used in previous meta-reviews 1 : Campbell Collaboration reviews published on or before May 2017 (n = 135) and used in Polanin and Nuijten 2,8 ; reviews published in the Review of Educational Research or Psychological Bulletin on or before May 2013 and used in Polanin and Pigott 5 (n = 137) 5 ; and 3 reviews on intelligence and IQ, found by searching the ISI Web of Knowledge and published on or before August 2014, used in Nuijten and colleagues (2018) 15 (n = 130). The results of using statcheck on this sample revealed that, of the 87 meta-analyses with NHST results reported in APA style in the full text, 39.1% contained at least one statistical inconsistency and 8% contained at least one gross inconsistency where the statistical conclusion may have changed.…”