Empirically Derived Guidelines for Effect Size Interpretation in Social Psychology

Abstract

A number of recent research publications have shown that commonly used guidelines for interpreting effect sizes suggested by Cohen (1988) do not fit well with the empirical distribution of those effect sizes, and tend to overestimate them in many research areas. This study proposes empirically derived guidelines for interpreting effect sizes for research in social psychology, based on analysis of the true distributions of the two types of effect size measures widely used in social psychology (correlation coefficient and standardized mean differences). Analysis was carried out on the empirical distribution of 9884 correlation coefficients and 3580 Hedges’ g statistics extracted from studies included in 98 published meta-analyses. The analysis reveals that the 25th, 50th, and 75th percentiles corresponded to correlation coefficients values of 0.12, 0.25, and 0.42 and to Hedges’ g values of 0.15, 0.38, and 0.69, respectively. This suggests that Cohen’s guidelines tend to overestimate medium and large effect sizes. It is recommended that correlation coefficients of 0.10, 0.25, and 0.40 and Hedges’ g of 0.15, 0.40, and 0.70 should be interpreted as small, medium, and large effects for studies in social psychology. The analysis also shows that more than half of all studies lack sufficient sample size to detect a medium effect. This paper reports the sample sizes required to achieve appropriate statistical power for the identification of small, medium, and large effects. This can be used for performing appropriately powered future studies when information about exact effect size is not available.

Effect Size Interpretation, Sample Size Calculation, and Statistical Power in Gerontology

“…research, but also stated that these should ultimately only be used when no specific information is available regarding the likelihood of various effect sizes. The results of the current study suggest that Cohen's (1988Cohen's ( , 1992 group difference effect sizes is very similar to those reported by Lovakov and Agadullina (2017;Hedges' g = 0.15, 0.38, and 0.69), although were lower than Quintana's (2017) estimates obtained in case-control studies of heart rate variability (Cohen's d = 0.26,0.51,and 0.88), possibly due to the wide range of research topics included in the current study.…”

“…These values have been rounded to the nearest 0.05 from the calculated percentiles (Table 1) for ease of use. It is likely that the observed estimates in the current study vary from Cohen's guidelines and previous research in other fields (e.g., Gignac & Szodorai, 2016;Lovakov & Agadullina, 2017;Quintana, 2017) for a couple of major reasons. First, it is possible that experimental methods used in gerontology may differ from other fields of research, such as how measurements of effects of interest are conducted, and potential between-participants variability with regards to outcomes of experimental manipulations and/or naturalistic observations in a representative sample of aging adults.…”

“…Gignac & Szodorai (2016) reported small, medium, and large correlations of .11, .19, and .29 in individual differences research in psychology, and Quintana (2017) observed Cohen's d values of 0.26, 0.51, and 0.88 for small, medium, and large effects in case-control studies of heart rate variability. Additionally, Lovakov & Agadullina (2017) reported Hedges' g values of 0.15,0.38,and 0.69,and Pearson's r values of .12,.25,and .42 in social psychology.…”

“…For example, if a social psychologist expects a medium effect size in a study examining differences between two groups (i.e., using an independent samples t-test), the required sample size to achieve power of .80 with alpha of .05 is n = 64 per group when using Cohen's (1988) estimate of Cohen's d = 0.50. When using Lovakov and Agadullina's (2017) estimate of Hedge's g = 0.38, however, 110 participants per group are required to achieve power of .80. Assuming the lower estimate is correct, a researcher would only achieve power of .57 if he/she recruited 64 participants per group, per calculations based on Cohen's estimate.…”

Are Immigration Policy Preferences Based on Accurate Stereotypes?

“…An accuracy correlation of r = .85 (r = .81 when including Syria) is quite remarkable, and is large even for aggregate-level data (though we are not aware of any formal effect size analyses or guidelines). Similarly, the median individual accuracy observed of r = .65 (r = .62 when including Syria) is likewise larger than the majority of effects observed in social science (Lovakov & Agadullina, 2017;Richard, Bond, & Stokes-Zoota, 2003).…”