How do neuropsychologists define cognitive constructs? Further thoughts on limitations of factor analysis used with normal or mixed clinical populations

Abstract

In a recent study, we empirically demonstrated limitations in traditional ways that psychologists have used factor analysis to define cognitive constructs (Delis et al., 2003). Our criticism of factor analysis was not directed at this statistical method per se, but rather at how it has often been employed by psychologists to test cognitive constructs. Specifically, we pointed out shortcomings in using this techniquewith normal or mixed clinical populations. We argued that the factor-analytic studies of memory tests with normal or mixed clinical populations often yielded solutions in which measures of immediate and delayed memory loaded on the same factor. This particular use of factor analysis can mask important distinctions between critical cognitive functions that have been demonstrated using other research methods, such as experimental manipulations or case studies. We then conducted a factor-analytic study that empirically demonstrated that, whereas immediate and delayed memory measures load on the same factor when using normal or mixed clinical samples, these measures load onseparatefactors when using a homogenous population of patients with Alzheimer's disease (AD). We drew the conclusion that factor analytic techniques can still be used asonemethod for exploring conceptual relationships, but only if these methods are used as part of a systematic, programmatic exploration involving separate confirmatory factor analyses using multiple homogenous patient populations. In the first published reply to our study, Larrabee (2003) pointed out other limitations, stating that application of factor-analytic techniques to a single test that yields multiple measures may result in global, simplistic solutions due to method variance, which is the tendency of different variables from the same test to correlate significantly.

The Uniform Dataset 3.0 Neuropsychological Battery: Factor Structure, Invariance Testing, and Demographically Adjusted Factor Score Calculation

“…On a related note, researchers and clinicians are cautioned against utilizing the factor scores in populations with known cognitive impairments. Indeed, concern has been raised in past research about our ability to generalize factor models from normal to clinical populations, as measurement properties may change in these different groups (Delis, Jacobson, Bondi, Hamilton, & Salmon, 2003;Jacobson, Delis, Hamilton, Bondi, & Salmon, 2004). Future research will examine the factor structure of the UDS3NB in clinical samples, as well as the utility of factor scores in these groups.…”

A Study on the Structure of Intelligence Measured by the K-WPPSI-IV

“…2 , 3 , 8-10 (Carroll, 1993(Carroll, , 2012Horn & Blankson, 2012;Jacobson et al, 2004). (Morgan, Rothlisberg, McIntoch, & Hunt, 2009;K.…”

Comparison of Wechsler Memory Scale–Fourth Edition (WMS–IV) and Third Edition (WMS–III) dimensional structures: Improved ability to evaluate auditory and visual constructs

“…A useful instrument should have an underlying structure that reflects diagnostically relevant constructs. However, in contrast to this position, based upon the results of numerous factor analytic studies that failed to differentiate between important immediate and delayed memory constructs, some researchers have suggested that factor analysis should not be implemented to evaluate memory instruments (e.g., see, Delis, Jacobson, Bondi, Hamilton, & Salmon, 2003;Jacobson, Delis, Hamilton, Bondi, & Salmon, 2004;Millis, Malina, Bowers, & Ricker, 1999). The failure of data reduction methods to differentiate between these constructs is related to significant shared variance between immediate and delayed memory tasks (i.e., efficient delayed memory is to a degree dependent upon intact immediate memory).Given the shared variance between immediate and delayed memory tasks, it is inappropriate to expect, and highly unlikely, that corresponding factors would be observed.…”