The quadratic relationship between difficulty of intelligence test items and their correlations with working memory

Abstract

Fluid intelligence (Gf) is a crucial cognitive ability that involves abstract reasoning in order to solve novel problems. Recent research demonstrated that Gf strongly depends on the individual effectiveness of working memory (WM). We investigated a popular claim that if the storage capacity underlay the WM–Gf correlation, then such a correlation should increase with an increasing number of items or rules (load) in a Gf-test. As often no such link is observed, on that basis the storage-capacity account is rejected, and alternative accounts of Gf (e.g., related to executive control or processing speed) are proposed. Using both analytical inference and numerical simulations, we demonstrated that the load-dependent change in correlation is primarily a function of the amount of floor/ceiling effect for particular items. Thus, the item-wise WM correlation of a Gf-test depends on its overall difficulty, and the difficulty distribution across its items. When the early test items yield huge ceiling, but the late items do not approach floor, that correlation will increase throughout the test. If the early items locate themselves between ceiling and floor, but the late items approach floor, the respective correlation will decrease. For a hallmark Gf-test, the Raven-test, whose items span from ceiling to floor, the quadratic relationship is expected, and it was shown empirically using a large sample and two types of WMC tasks. In consequence, no changes in correlation due to varying WM/Gf load, or lack of them, can yield an argument for or against any theory of WM/Gf. Moreover, as the mathematical properties of the correlation formula make it relatively immune to ceiling/floor effects for overall moderate correlations, only minor changes (if any) in the WM–Gf correlation should be expected for many psychological tests.

Intelligence test items varying in capacity demands cannot be used to test the causality of working memory capacity for fluid intelligence

“…µWMC -Nrules = 0), and the shaded lines represents the average pattern of correlations for different levels of correlations between WMC and the ability θ underlying intelligence test performance fitted with a locally weighted least squares regression. rows) the results of our simulations replicated the result ofSmoleń and Chuderski (2015):…”

“…Both analytical derivations and empirical results of their study showed that, depending on the size of the correlation between two variables and the amount of floor or ceiling effects for specific items, correlations can increase, be almost stable, or decrease across items varying in their mean error rate. On this basis, Smoleń and Chuderski (2015) conclude that results from such analyses cannot be informative regarding the capacity hypothesis. Here we present a conceptual analysis of the capacity hypothesis and its implications for WMC-intelligence correlations, which arrives at the same conclusion.…”

“…A methodological critique of these types of analyses (Smoleń & Chuderski, 2015) argued that "although intuitively attractive" the underlying idea is "fundamentally flawed". Specifically, Smoleń and Chuderski (2015) proposed that the pattern of correlations across items varying in difficulty (i.e. mean error rates) necessarily follows a quadratic function that is tied to the amount of floor and ceiling effects for the different items.…”

Intelligence test items varying in capacity demands cannot be used to test the causality of working memory capacity for fluid intelligence

“…In addition, the line shading in this plot illustrates that the correlation of item performance with WMC increases the stronger WMC is correlated with the ability θ underlying intelligence test performance. 3 Finally, we compared our results to the results of Smoleń and Chuderski (2015) indicating that the correlation of item performance with WMC follows a quadratic pattern due to floor and ceiling effects. We plotted the correlations as a function of the proportion of simulated subjects that were able to solve an item (i.e., the item difficulty p in classical test theory).…”

An application of the Linear Logistic Test Model for the construction of a Fluid Intelligence Test

“…The present findings are also highly relevant in the domain of education. First, it is well known that fluid intelligence predicts many forms of academic and school achievements and that it correlates with a good management in daily life (Deary, 2012;Smolen and Chuderski, 2015). Therefore, a better understanding of the specific mechanisms involved in fluid intelligence will inform educators about the sources of variation for people in academic contexts, thereby providing a broad view for developing more opportune strategies for teaching and evaluation.…”