How Accurate Is Your Correlation? Different Methods Derive Different Results and Different Interpretations

Abstract

Assessing the association between conceptual constructs are at the heart of quantitative research in educational and psychological research. Researchers apply different methods to the data to obtain results about the correlation between a set of variables. However, the question remains, how accurate are the results of the correlation obtained from these methods? Although various considerations should be taken to ensure accurate results, we focus on the types of analysis researchers apply to the data and discuss three methods most researchers use to obtain results about correlation. Particularly, we show how correlation results in bivariate correlation, confirmatory factor analysis (CFA), and exploratory structural equation modeling (ESEM) differ substantially in size. We observe that methods that assume independence of the items often generate inflated factor correlations whereas methods that relax this assumption present uninflated, thus more accurate correlations. Because factor correlations are inflated in bivariate correlation and CFA, the discriminant validity of the constructs is often unattainable. In these methods, the size of the correlation can be very large and biased. We discuss the reasons for this variation and suggest the type of correlation that researchers should select and report.

The role of basic psychological needs on volunteering and national responsibility during the COVID-19 pandemic: Results from the context of Saudi Arabia

“…The model fits the data adequately [χ 2 = 5.59, df = 3, p < 0.001, SRMR = 0.04, RMSEA = 0.07, RMSEA Low and Hi 95% CI (0.01, 0.15), CFI = 0.99, TLI = 0.95, IFI = 1.00]. The results of the path model illustrated that the correlation between the predictors was within the acceptable levels (Alamer and Marsh, 2022;Shao et al, 2022). The variance explained by the predictor variables in the outcome variables, i.e., sense of national responsibility, searching for opportunities to volunteer, and the quantity of volunteering made, was 23, 37, and 08%, respectively.…”

Construct validation of the revised Metacognitive Awareness of Reading Strategies Inventory (MARSI-R) and its relation to learning effort and reading achievement

“…If such misspecified model is assumed, researchers are often obliged to apply data-driven modifications to compensate for ill-fitted models (Alamer, 2022c;Alamer & Marsh, 2022;Alrabai & Alamer, 2022). Nonetheless, procedures such as multiple removal of items and multiple correlation of error terms negatively affect the quality of the internal structure of the models, which consequently provide questionable evidence of the construct validity of the instrument under assessment (Shao et al, 2022). We encourage researchers in metacognitive reading strategies in general to consider composite models in their analysis.…”

A cross-lagged panel analysis of second language achievement and enjoyment

“…Furthermore, the linear trend analysis of Botes et al (2020) indicated that increased self-perceived FL proficiency resulted in significantly increased FLE [ F (1, 1,617) = 58.867; p < 0.001]. However, the statistical methods adopted in most previous studies, such as correlation and regression analyses, t -tests, and ANOVA, cannot establish causality ( Shao et al, 2022 ). Similarly, the cross-sectional design of studies conducting path analysis made it difficult to determine causality ( Collier, 2020 ), as did the non-experimental design of Dewaele et al (2018) .…”