Meta-analysis of correlation coefficients: A cautionary tale on treating measurement error.

Abstract

A scale to measure a psychological construct is subject to measurement error. When meta-analyzing correlations obtained from scale scores, many researchers recommend correcting for measurement error. I considered three caveats when correcting for measurement error in meta-analysis of correlations: (a) the distribution of true scores can be non-normal, resulting in violation of the normality assumption for raw correlations and Fisher's z transformed correlations; (b) coefficient alpha is often used as the reliability, but correlations corrected for measurement error using alpha can be inaccurate when some assumptions of alpha (e.g., tau-equivalence) are violated; and (c) item scores are often ordinal, making the disattenuation formula potentially problematic. Via three simulation studies, I examined the performance of two meta-analysis approaches-with raw correlations and z scores. In terms of estimation accuracy and coverage probability of the mean correlation, results showed that (a) considering the truescore distribution alone, estimation of the mean correlation was slightly worse when true scores of the constructs were skewed rather than normal; (b) when the tau-equivalence assumption was violated and coefficient alpha was used for correcting measurement error, the mean correlation estimates can be biased and coverage probabilities can be low; and (c) discretization of continuous items can result in biased estimates and undercoverage of the mean correlations even when tau-equivalence was satisfied. With more categories and/or items on a scale, results can improve whether tau-equivalence was met or not. Based on these findings, I gave recommendations for conducting meta-analyses of correlations.

Overprotective parenting experiences and early maladaptive schemas in adolescence and adulthood: A systematic review and meta‐analysis

“…Another potential limitation is that the correlations in our meta‐analyses were not adjusted for the unreliability of the variables. Zhang ( 2021 ) recommended that meta‐analyses with continuous variables should correct for coefficient alpha to obtain an accurate mean effect size estimate, corrected for measurement error. However, Meta‐Essentials does not provide this functionality, adjusting effect sizes upwards can lead to unjustifiably large mean effect sizes, and not all included studies reported on the reliability of their measures Lipsey and Wilson ( 2001 ).…”

The higher-order structure of early maladaptive schemas: A meta-analytical approach

“…Pooled correlations were obtained for the total sample and separately for the clinical and non-clinical subsamples. Deviating from the preregistered protocol, it was not adjusted for score unreliability because reliability is usually reported as Cronbach's alpha, but important assumptions (e.g., tau-equivalence) for using alpha are often not met ( 38 ).…”

Correcting for measurement error under meta‐analysis of z‐transformed correlations

“…Provided accurate transformation from the mean z-transformed correlation estimates and the associated confidence intervals to the estimates and confidence intervals of (Hafdahl, 2009;Law, 1995), meta-analysis using both the raw correlations and z-transformed correlations will yield reasonably accurate estimates and inferences of . Extending from these former studies, Zhang (2022) examined these two approaches with non-normal X and Y scores and no artefacts in one of the simulations and found that they still yielded in general reasonably accurate estimates and good coverage rates for .…”

“…More relevantly, Hedges (1988) discussed the correction of both z-transformed correlation estimates and within-study sampling error variance to estimate and statistically infer . 1 In their major simulation studies, Zhang (2022) examined the performance of meta-analyses of z-transformed correlations wherein individual z-transformed correlations were corrected for measurement error using coefficient alpha based on Hedges (1988, Eq. 5) and 1∕(N i − 3) was used as the within-study sampling error variance estimate.…”