Effect Sizes, Power, and Biases in Intelligence Research: A Meta-Meta-Analysis

Abstract

In this meta-study, we analyzed 2442 effect sizes from 131 meta-analyses in intelligence research, published from 1984 to 2014, to estimate the average effect size, median power, and evidence for bias. We found that the average effect size in intelligence research was a Pearson’s correlation of 0.26, and the median sample size was 60. Furthermore, across primary studies, we found a median power of 11.9% to detect a small effect, 54.5% to detect a medium effect, and 93.9% to detect a large effect. We documented differences in average effect size and median estimated power between different types of intelligence studies (correlational studies, studies of group differences, experiments, toxicology, and behavior genetics). On average, across all meta-analyses (but not in every meta-analysis), we found evidence for small-study effects, potentially indicating publication bias and overestimated effects. We found no differences in small-study effects between different study types. We also found no convincing evidence for the decline effect, US effect, or citation bias across meta-analyses. We concluded that intelligence research does show signs of low power and publication bias, but that these problems seem less severe than in many other scientific fields.

Distinguishing specific from general effects in cognition research.

“…Next, we discuss the feasibility of using a latent variable model instead of traditional ANOVA methods, when the sample sizes per group are small (N = 50 to N = 100), and the expected effect size is small to moderate (Cohen's d = 0.40). These estimates coincide with the median sample size (N = 60) and median weighted effect size (r = 0.26) found in approximately 2400 effect sizes in 131 meta-analyses in intelligence research (Nuijten, van Assen, Augusteijn, Crompvoets, & Wicherts, 2019).…”

Using and Understanding Power in Psychological Research: A Survey Study

“…For instance, an examination of 2261 psychology papers by Szucs and Ioannidis (2017) found that the mean power for detecting small effects in psychology was 23%, 60% for medium effects, and 78% for large effects, while Stanley et al's (2018) review of 200 meta-analyses calculated a median overall statistical power of only 36%. Most recently, Nuijten et al (2020) found that, in intelligence research, studies only had 11.9% power to detect small effects, along with a median sample size of just 60 participants. It appears that several decades after Cohen's first review, very few studies still come close to the widely accepted minimum power level of 80%.…”

Verifying the accuracy of statistical significance testing in Campbell Collaboration systematic reviews through the use of the R package statcheck

“…The second sample derived from a review of the literature about intelligence and IQ (Nuijten, Van Assen, Augusteijn, Crompvoets, & Wicherts, 2018),in which the authors analyzed meta‐analyses found in ISI Web of Science as of August 2014 to estimate the typical effect size, the median power, and patterns of biases. They included quantitative meta‐analyses with complete data tables with sufficient information to calculate or retrieve primary study effect size and standard error.…”