A machine learning approach for the factorization of psychometric data with application to the Delis Kaplan Executive Function System

Abstract

While a replicability crisis has shaken psychological sciences, the replicability of multivariate approaches for psychometric data factorization has received little attention. In particular, Exploratory Factor Analysis (EFA) is frequently promoted as the gold standard in psychological sciences. However, the application of EFA to executive functioning, a core concept in psychology and cognitive neuroscience, has led to divergent conceptual models. This heterogeneity severely limits the generalizability and replicability of findings. To tackle this issue, in this study, we propose to capitalize on a machine learning approach, OPNMF (Orthonormal Projective Non-Negative Factorization), and leverage internal cross-validation to promote generalizability to an independent dataset. We examined its application on the scores of 334 adults at the Delis–Kaplan Executive Function System (D-KEFS), while comparing to standard EFA and Principal Component Analysis (PCA). We further evaluated the replicability of the derived factorization across specific gender and age subsamples. Overall, OPNMF and PCA both converge towards a two-factor model as the best data-fit model. The derived factorization suggests a division between low-level and high-level executive functioning measures, a model further supported in subsamples. In contrast, EFA, highlighted a five-factor model which reflects the segregation of the D-KEFS battery into its main tasks while still clustering higher-level tasks together. However, this model was poorly supported in the subsamples. Thus, the parsimonious two-factors model revealed by OPNMF encompasses the more complex factorization yielded by EFA while enjoying higher generalizability. Hence, OPNMF provides a conceptually meaningful, technically robust, and generalizable factorization for psychometric tools.

Construct identification in the neuropsychological battery: What are we measuring?

“…In contrast, other investigators identified two-factor solutions that essentially separated timed tasks from abstraction tests. Our analysis would be unlikely to identify an “abstraction” factor given we did not include Word Context, Twenty Questions, Proverb, Sorting, or Tower Tests (Camilleri et al, 2021; Savla et al, 2012). Our solution is also simpler than a model that emphasized Cattell-Horn-Carroll (CHC) theory and had representations of other tasks that likely highlighted nonexecutive cognitive abilities (Floyd et al, 2010).…”

“…We had more than 100 cases each on the D–KEFS Trail Making, VF, and CWIT. There have been several factor analytic studies of the D–KEFS (Camilleri et al, 2021; Floyd et al, 2006; Karr et al, 2019; Latzman & Markon, 2010). Latzman and Markon (2010) used exploratory factor models to identify D–KEFS structure.…”

“…Our analyses were therefore limited to analysis of VF, TMT, and CWIT scores. The results from Camilleri et al (2021), who used exploratory factor analysis (EFA) and machine learning approaches to examine D-KEFS structure, would suggest that we might expect a two-factor solution based on these tests, with one factor reflecting "Inhibition" and loading on both CWIT and TMT and another "Fluency" factor. Savla and colleagues (Savla et al, 2012) also found EFA to yield a two-factor solution with "Flexibility" and "Abstraction" components, but we do not include any of the tests that loaded heavily on their Abstraction factor.…”

The latent structure of the Delis–Kaplan system for autism

“…While the authors of the D‐KEFS do not explicitly state its proposed latent structure, previous studies have found a three‐factor structure to be a good fit in nonautistic samples (Karr et al, 2019; Latzman & Markon, 2010), although there is disagreement across studies as to what the latent factors are. Principal component analysis of a sample of nonautistic adults found a two‐factor model differentiating between simple and complex EF tasks (Camilleri et al, 2021). In a sample of adults with schizophrenia, an exploratory factor analysis found a two‐factor (CF and Abstraction) model to be a good fit for the data (Savla et al, 2012).…”

Development of Executive Functions' Structure in 8-12 Year Olds

“…These mixed findings are to be considered in the context of methodological differences across studies, including large variations in executive functions' measurement (Yangüez et al, 2023). Such measurement heterogeneity coupled with the use of underpowered samples likely contribute to the replicability issue the field faces (Camilleri et al, 2021;Karr et al, 2018). Relying on recent methodological work about the psychometric modeling of executive functions (Yangüez et al, 2023), the present study aims to extend our understanding of developmental changes in executive functions' structure during the pivotal age of middle childhood (i.e., period from 8 to 12 years old).…”