Increasing the Reliability of Ability‐Achievement Difference Scores: An Example Using the Kaufman Assessment Battery for Children

Abstract

In this study, we focused on increasing the reliability of ability‐achievement difference scores using the Kaufman Assessment Battery for Children (KABC) as an example. Ability‐achievement difference scores are often used as indicators of learning disabilities, but when they are derived from traditional equally weighted ability and achievement scores, they have suboptimal psychometric properties because of the high correlations between the scores. As an alternative to equally weighted difference scores, we examined an orthogonal reliable component analysis, (RCA) solution and an oblique principal component analysis (PCA) solution for the standardization sample of the KABC (among 5‐ to 12‐year‐olds). The components were easily identifiable as the simultaneous processing, sequential processing, and achievement constructs assessed by the KABC. As judged via the score intercorrelations, all three types of scores had adequate convergent validity, while the orthogonal RCA scores had superior discriminant validity, followed by the oblique PCA scores. Differences between the orthogonal RCA scores were more reliable than differences between the oblique PCA scores, which were in turn more reliable than differences between the traditional equally weighted scores. The increased reliability with which the KABC differences are assessed with the orthogonal RCA method has important practical implications, including narrower confidence intervals around difference scores used in individual administrations of the KABC.

Increasing the reliability of ipsative interpretations in neuropsychology: A comparison of reliable components analysis and other factor analytic methods

“…This result may be specific to the present data or may indicate that RCA solutions are more sensitive than other methods to small perturbations in observed correlations. Findings also contradicted previous studies suggesting that RCA produces similar factors to other methods (Caruso & Cliff, 2000;Caruso & Witkiewitz, 2002); however this was only the case for the last factor extracted. For PCA and PAF solutions, the last factor was primarily made up of variables with low reliability (WCST-perseverative errors r xx 5 .64; WCST-categories r xx 5 .70), but for the RCA-varimax solution the composition of this factor emphasized more reliable variables (WAIS-III POI r xx 5 .93; Boston Naming Test r xx 5 .93).…”

“…These studies have found that RCA produces a similar factor 2 structure to other exploratory factor analytic techniques, but that RCA based composites yield more reliable cognitive difference scores than other techniques. Recent findings from a study by Caruso and Witkiewitz (2002) found that the principle method by which RCA generated reliable difference scores was through the creation of orthogonal composites using varimax rotation of factor weights. This suggests that the uncorrelated nature of factors derived using varimax rotation, regardless of the specific extraction, may be the primary driving force in the higher reliability of difference scores derived from factor analytic methods.…”

Measurement error: implications for diagnosis and discrepancy models of developmental dyslexia

“…where z r xx is the reliability of the intelligence test, r yy is the reliability of the reading test, and r xy is the correlation between the two tests (Caruso & Witkiewitz, 2002;Leeb & Weinberg, 1977;Overall & Woodward, 1975;Schulte & Borich, 1984). Using this formula, one would find that the higher the correlation between intelligence and reading tests, then the lower the reliability of the difference score (Caruso & Witkiewitz, 2002;Nunnally & Bernstein, 1994;Zimmerman, 1994).…”

“…Using this formula, one would find that the higher the correlation between intelligence and reading tests, then the lower the reliability of the difference score (Caruso & Witkiewitz, 2002;Nunnally & Bernstein, 1994;Zimmerman, 1994).…”

“…Second, the standard deviation of the difference score (S.D. d ) needs to be calculated using the formula specified by Caruso and Witkiewitz (2002):…”