Theory by the numbers? Some concerns about meta‐analysis as a theoretical tool

Abstract

It is with considerable trepidation and some real ambivalence that I find myself responding to Mike Pressley's request for a comment on Tang and Hall's (1995) meta-analysis on the literature on overjustification. In fact, I feel a bit like someone who, though known as a lover of cocker spaniels, fox terriers, and miniature collies, has somehow been asked to serve as a judge at the State pit bull and Rottweiler competition. Such a judge might have little love for these breeds; indeed, he or she might believe them to be profoundly dangerous and a downright menace to society. None the less, he or she might still feel capable of distinguishing the best from the worst of each.Let me lay my cards clearly on the table. I am not a great fan of meta-analysis as a theoretical tool. Obviously, there are a variety of purely empirical comparisons and existence claims for which the use of meta-analytic procedures to combine the results of multiple studies may be useful and important. In the medical arena, for example, one might often want to test whether seemingly consistent, but small and rarely individually significant, differences that appear in multiple studies (such as the detrimental health effects of exposure to low levels of lead contamination or second-hand smoke) represent 'real' phenomena warranting attention or intervention. I am not at all convinced, however, that these same techniques have particular value as tools for reviewing diverse theoretical literatures in which the simple 'null hypothesis' that there is no effect is patently false and the relevant question is to understand the conditions under which a particular effect is likely or not likely to occur-especially when those literatures include findings derived from a variety of different and distinct experimental procedures and measures.At the same time, within this last genre, I do find Tang and Hall's (1995) use of meta-analysis to try to test theoretical generalizations concerning the overjustification The author would like to thank Of course, one theoretically possible solution to these sorts of problems, given a very large sample of studies on a particular topic, would be to subdivide that sample on the basis of differences in procedures, measures, and the like until only relatively homogenous clusters of studies remained. In practice the problem, of course, is that there are rarely enough studies to permit this sort of differentiation and yet retain sufficient statistical power for comparing and contrasting clusters. Rather, as in Tang and Hall's (1995) analyses, such subsamples rapidly result in such small numbers of cases as to make true comparative analyses impractical.

Null hypothesis significance testing: A review of an old and continuing controversy.

“…We need to also bear in mind that it is possible for knowledge to cumulate without the aid of specific meta-analytic techniques; it has done so in the physical sciences quite effectively for a long time. Moreover, meta-analysis is not entirely free of problems and offers its own set of possibilities for misuse (Chow, 1996; Erwin, 1997; Gallo, 1978; Knight, Fabes, & Higgins, 1996; Lepper, 1995; Leviton & Cook, 1981; Shapiro, 1997; Sohn, 1996; G. T. Wilson & Rachman, 1983).…”

Understanding the effects of extrinsic rewards on intrinsic motivation—Uses and abuses of meta-anlysis: Comment on Deci, Koestner, and Ryan (1999).

“…By contrast, when evaluating the predictive power, or functional significance, of an experimental effect as psychologists, rather than as statisticians, researchers typically use a very different metric. In this more psychological sense of the term, the functional "effect size" and the consequent inferential power of a given comparison depend not only on the statistical significance of the difference between two conditions and the size of the samples but also on several additional factors: (a) the strength of the manipulation producing that difference, (b) the sensitivity and consequentiality of the dependent measures, and (c) the range and power of factors that were not explicitly controlled and thereby constitute the natural error variance against which the effect must "compete" for significance (see, e.g., Abelson, 1995;Lepper, 1995;Prentice & Miller, 1992;L. Ross & Nisbett, 1991).…”

“…In addition to the foregoing problems that may result from the application of meta-analytic techniques to complex theoretically derived literatures, other issues can arise when purely statistical effect-size measures are used to compare studies that vary greatly in their procedures and measures. In certain literatures in particular, measures of purely statistical effect size may not be good indicators of functional or psychological effect size, and in fact these two measures can be negatively correlated (Lepper, 1995). Again, greater attention to the social psychology of psychological research may help in understanding these additional problems with the unthinking use of meta-analytic techniques.…”

“…Consider two different meanings of effect size (Lepper, 1995; Lepper et al, 1996; Prentice & Miller, 1992). On the one hand, purely statistical measures of effect size depend solely on the means, standard deviations, and sample sizes in the relevant comparison conditions.…”

Meta-Analysis and the Development of Knowledge

“…Even though meta-analysis enjoys widespread application, researchers have differing opinions about the ultimate purpose of meta-analysis (Hunter & Schmidt, 2004)—especially with regard to the role of meta-analysis in the advancement of knowledge. Some researchers argue that meta-analysis merely provides a description and summary review of extant studies in a field (e.g., Glass, 1976; Glass et al, 1981; Guzzo et al, 1987; Hoyle, 1993; Sohn, 1996) but is not ideally suited for theory development and scientific discovery (e.g., Chow, 1987; Guzzo et al, 1987; Hoyle, 1993; Lepper, 1995; Sohn, 1995, 1996). For example, questions are raised on whether a meta-analysis exerts a “chilling effect” on the number of primary studies conducted after it has been published (Slavin, 1984).…”