Cognitive profile analysis: A shared professional myth.

Abstract

Interpretation of cognitive test profiles is popular in school psychological training and practice but is not supported by research evidence. This special issue contains four studies that provide additional data regarding cognitive test profiles. Consistent with the largely negative research literature, detailed analysis found all four cognitive profile reports lacking reliability, validity, or diagnostic utility; even cognitive profiles composed of composites were psychometrically weak. These results were not surprising because ipsative methods are inferior to normative methods in cognitive assessment. Given this consistent failure of empirical validation, belief in the utility of cognitive test profile interpretations was likened to a shared professional myth and it was recommended that psychologists eschew the application of cognitive test profiles for differential diagnosis and remediation. Approximately 74% of school psychology training programs place moderate to great emphasis on the use of subtest scores in their individual cognitive assessment courses (Alfonso, Oakland, LaRocca, & Spanakos, 2000) and school psychologists frequently analyze cognitive subtest profiles (Pfeiffer, Reddy, Kletzel, Schmelzer, & Boyer, 2000). Among their sample of practitioners, for example, Pfeiffer et al. (2000) found that almost 70% reported factor scores or profile analysis to be a useful feature of the Wechsler Intelligence Scale for Children-Third Edition (WISC-III) and 29% reported that they derived specific value from individual WISC-III subtests. Based on their survey results, Pfeiffer et al. (2000) suggested that critics of cognitive subtest analysis "extend an invitation to the practitioner to learn why practitioners-operating in the complexity of the clinical world-find considerable value in using clinical interpretative techniques such as profile analysis" (p. 384). However, personal explanations are an umeliable validation strategy (Ruscio, 1998) and neither popularity nor longevity of a clinical practice necessarily speaks to its verity. In medicine, for example, practices widespread for centuries were later recognized to have little or no curative effect. Medicine, therefore, provides dramatic examples "of the inadequacies of clinical observation when compared to scientific method" (Ziskin, 1995, p. 53

Estimating Cognitive Profiles Using Profile Analysis via Multidimensional Scaling (PAMS)

“…It also has been argued that profile analysis is only useful when recommendations are made under the assumption that probabilities of producing outcomes are known (Carroll, 2000). According to these criteria, however, the previous literature does not contain impressive support for the utility of profile analysis for individual test-takers (Watkins, 2000). In the cognitive testing domain, research may not have yielded fruitful results about differential diagnoses for clinical utility because the substantial g factor (or general ability) is confounded with group factors.…”

Research into practice: Cattell‐Horn‐Carroll cognitive assessment in practice: Eligibility and program development issues

“…However, as McGrew and colleagues (1997) later point out, most of the anti-specific ability research in school psychology has been conducted with measures that are based on an outdated conceptualization of intelligence (viz., Wechsler batteries) and have used research methods that have placed primary emphasis on prediction with little attention to explanation and theoretical understanding of the relations between general and specific cognitive abilities and school achievement. (p. 191; italics in original) In addition, the methodology used by McDermott and his colleagues (e.g., Glutting, McDermott, Konold, Snelbaker, & Watkins, 1998;Glutting, Youngstrom, Ward, Ward, & Hale, 1997;McDermott et al, 1990;Watkins, 2000) fails to take into account the multicollinearity of measures of cognitive functioning (e.g., see Hale, Fiorello, Kavanaugh, Hoeppner, & Gaither, 2001). Structural equation modeling (SEM), used in many of the studies presented here, provides a more accurate view of the contributions of g and multiple cognitive abilities to academic achievement.…”

Interpreting Intelligence Test Results for Children with Disabilities: Is Global Intelligence Relevant?

“…Typically citing early studies that deny the utility of aptitude-treatment interactions (e.g., Cronbach, 1975;Ysseldyke & Sabatino, 1973), advocates of global IQ interpretation argue against intelligence subtest interpretation or profile analysis, claiming the practice is based more on fiction than fact (McDermott, Fantuzzo, & Glutting, 1990Watkins, 2000). Global IQ advocates suggest there are no definitive studies that support subtest or factor profile analysis, and that global composites have well-documented predictive validity (Glutting, McDermott, Konold, Snelbaker, & Watkins, 1998;Glutting, Youngstrom, Ward, Ward, & Hale, 1997).…”