A Tree‐based Approach to Proficiency Scaling

Abstract

The process of translating examinees' observed test results into probabilistic statements about the patterns of skill mastery associated with performance at successive points on a test's reported score scale has been termed proficiency scaling. This paper introduces a new approach to proficiency scaling which was specifically developed to provide accurate, instructionally-relevant score interpretations, even when the test under consideration is a nondiagnostic, broad-based ability test such as the SAT. The approach employs techniques derived from the realm of tree-based regression to classify test items into nonoverlapping clusters such that all of the items in each cluster have similar values of item difficulty and require similar combinations of skills. A nonparametric smoothing procedure is then used to summarize cluster performance conditional on underlying test score. The resulting cluster characteristic curves provide the probability of responding correctly to items with specified skill requirements, expressed as a function of the underlying test score. This new approach is described in the context of a specific application: developing proficiency interpretations for the reading comprehension subsection of the SAT I Verbal Reasoning Test. It is shown that the generated proficiency interpretations account for 91 % of the variation which would be accounted for by an optimal clustering solution. These results suggest that the approach is capable of extracting useful information about the specific combinations of skills needed to perform at increasingly higher levels on a test's reported score scale.

A principled design framework for college admissions tests: An affirming research agenda.

“…In response to this challenge, a number of new and promising psychometric approaches with a decidedly cognitive flavor are being developed. These include, for example, latent trait models (Embretson, 1985(Embretson, , 1994Samejima, 1995), statistical pattern classification methods (Sheehan, 1996;Tatsuoka, 1985Tatsuoka, , 1990, and causal probabilistic networks (Mislevy, 1995). As noted earlier, these approaches do not aim simply to rank students along a dimension of proficiency.…”

Construct Validity and Cognitive Diagnostic Assessment

“…Instead of inferring a general response tendency or behavior consistency of an examinee over a target domain of measurement, diagnostic assessment results provide a detailed account of the underlying cognitive basis of the examinee's performance by mining the richer information that is afforded by specific response patterns. Sophisticated measurement procedures, such as the rule-space methodology (Tatsuoka, 1995), the attribute hierarchy method (Leighton, Cierl, & Hunka, 2004), the tree-based regression approach (Sheehan, 1997a(Sheehan, , 1997b, and the knowledge space theory (Doignon & Falmagn . e, 1999), as well as specially parameterized psychometric models (De La Torre & Douglas, 2004;DiBello, Stout, & Roussos, 1995;Draney, Pirolli, & Wilson, 1995;Hartz, 2002;Junker & Sijtsma, 2001;Maris, 1999), have been developed for inferring diagnostic information.…”