What's Wrong with Research Literatures? And how to Make Them Right

Abstract

Meta-analysis is now the accepted procedure for summarizing research literatures in areas of applied psychology. Because of the bias for publishing statistically significant findings, while usually rejecting nonsignificant results, our research literatures yield misleading answers to important quantitative questions (e.g., How much better is the average psychotherapy patient relative to a comparable group of untreated controls? How much more aggressive are children who watch a great deal of violent TV than children who watch little or no violence on TV?). While all such research literatures provide overly optimistic meta-analytic estimates, exactly how practically important are these overestimates? Three studies testing the literature on implementation intentions finds only slightly elevated effectiveness estimates. Conversely, in three studies another growing research literature (the efficacy of remote intercessory prayer) is found to be misleading and is in all likelihood not a real effect (i.e., our three studies suggest the literature likely consists of Type I errors). Rules of thumb to predict which research literatures are likely invalid are offered. Finally, revised publication and data analysis procedures to generate unbiased research literatures in the future are examined.

Reducing procrastination using a smartphone-based treatment program: A randomized controlled pilot study

“…Major limitations of this study include: (a) the use of a small sample size, (b) absence of a sham control condition, (c) the absence of a MT-PRO only group that does not include face-to-face psychoeducation, and (d) the absence of experimental manipulation of potential mechanisms of change. Given the small sample size, results of this study should be interpreted with caution as the small and homogenous sample may have led to an overestimation of effect sizes ( Howard et al, 2009 , Ioannidis, 2008 ). Future studies should use larger and more heterogeneous samples.…”

Gender and Mathematics Education: An Overview

“…In summary: after four decades of consistent, persistent, and often insightful research on gender and mathematics there seems to be at best limited consensus on the size and direction of gender differences in mathematics performance. Might the tendency for statistically significant results to be accepted for publication while non-significant findings are rejected (as discussed by, e.g., Howard et al 2009) perhaps influence this summation? That there is great variation in the explanations put forward to account for any gender differences found is widely acknowledged.…”

“…Where should new research efforts be directed to ensure that the field will continue to advance and develop? According to Howard et al (2009):…”

A Meta-Analytic Review of Stand-Alone Interventions to Improve Body Image

“…Related to publication bias is the phenomenon of small sample bias : the tendency for estimates of the intervention effect to be more favourable in smaller studies. Coyne, Thombs, and Hagedoorn [ 54 ] recently critiqued interventions in the field of behavioural medicine for over-relying on small, underpowered trials [ 55 , 56 ]. Coyne et al [ 54 ] recommended that meta-analysts correct for small sample bias by estimating intervention effects separately for studies that contain at least 35 participants per cell, and thus have ≥ 55% power to detect an effect of medium magnitude.…”