Measures of explained variance: Cautions and limitations.

Abstract

Measures of explained variance (e.g., proportion of variance accounted for) are often considered to indicate the importance of a statistical finding. Three potential limitations to this viewpoint are discussed: psychometric, methodological, and theoretical. A psychometric perspective suggests that errors of measurement produce an upper bound to any measure of explained variance, this upper bound being the product of the reliabilities of the variables whose association is under investigation. A methodological perspective suggests several factors that influence the magnitude of measures of explained variance, including the intentions of the researcher, the design of the research, and the population sampled. A theoretical perspective suggests that most behavior has multiple determinants, and thus the magnitude of measures of explained variance is limited by nature. It is concluded that optimal use of measures of explained variance can be made when researchers examine the agreement between the magnitude that their theory would suggest and the empirical finding.

The effects of race/ethnicity, income, and family structure on adolescent risk behaviors

“…Rather, the low R 2 indicates tremendous heterogeneity of outcomes within racial/ethnic groups, income strata, and family structures. 49,50 Perhaps one of the factors that has made it difficult to address adolescent health problems in the United States is the focus on describing the problem through highlighting group differences and, when differences are found, concluding that we understand the primary contributing factors. In so doing, we run a high risk of building our interventions on variables that are not amenable to change and that, even if they were amenable to change, would not significantly alter behavioral outcomes.…”

Measures of Effect Size for Comparative Studies: Applications, Interpretations, and Limitations

“…But several critics have pointed out that measures of association are affected by the reliability of the measures, the heterogeneity of the populations being compared, the specific levels of the variables studied, the strength of the treatments, and the range of treatments (Maxwell et al, 1981;O'Grady, 1982;Sechrest & Yeaton, 1982), thus making such comparisons hazardous. Fern and Monroe (1996) presented an especially comprehensive discussion of these factors.…”

“…Over the years, several papers have been published tracing the historical developments of effect size measures (Dwyer, 1974;Glass & Hakstian, 1969;Maxwell, Camp, & Arvey, 1981;Richardson, 1996), suggesting alternative indices (Friedman, 1968;McGraw & Wong, 1992;Rosenthal & Rubin, 1982), and offering cautionary notes on the interpretation of these indices (Mitchell & Hartmann, 1981;Muray & Dosser, 1987;O'Grady, 1982;Sechrest & Yeaton, 1982;Strube, 1988). Textbooks on statistical methods have also included procedures for computing a number of indices of effect (e.g., Huberty, 1994;Keppel, 1991;Maxwell & Delaney, 1990;Stevens, 1996).…”

Errors and mistakes: Evaluating the accuracy of social judgment.

“…The usual practice with such correlations is to square measured did not resemble any real-life situations in which the them and thereby compute that somewhere between 9% and informants could have seen the subjects and (b) in the latter two 16% of the variance has been "accounted for." Such phrases, studies several years had elapsed between the behavior and the although not completely self-explanatory (O'Grady, 1982), do personality judgment.…”