Vertical Articulation of Cut Scores Across the Grades: Current Practices and Methodological Implications in the Light of the Next Generation of K–12 Assessments

Abstract

Federal accountability requirements after the No Child Left Behind (NCLB) Act of 2001 and the need to report progress for various disaggregated subgroups of students meant that the methods used to set and articulate performance standards across the grades must be revisited. Several solutions that involve either a priori deliberations or post‐hoc adjustments have been offered over the years. In this paper, I provide a methodological review of the alternative cut‐score articulation methods, including some novel solutions (e.g., using predictive methods) that have been proposed in the context of the next‐generation K–12 assessments. In systematically evaluating these methods, I focus on the psychometric challenges they might present and the practical feasibility of their operational implementation. In addition, results from a survey of several state departments of education help to provide information on the prevalence of these methods across the states. Overall, this review shows that none of the alternative methods is completely free of limitation; yet, each method provides solutions that are appropriate for addressing certain methodological and practical requirements. Therefore, in the context of the next‐generation assessments and the need to identify students who are on track to being college and career ready, practitioners are advised to consider a combination of methods and cautioned against overreliance on any single method.

Articulation of Cut Scores in the Context of the Next‐Generation Assessments

“…Ever since the initial push for reporting adequate yearly progress under NCLB legislation, several alternative cut‐score articulation methods (e.g., statistical interpolation, impact percentage smoothing) have been offered (see Cizek & Agger, ; Kannan, ). In addition to statistical methods used to articulate cut scores, in practice, many states use a vertical articulation or smoothing committee, composed of panelists from grade‐level standard‐setting committees, to review the grade‐level standards.…”

“…Use of vertical scales in developing cross‐grade tests can provide an ideal solution for articulation of performance standards and offer several practical benefits (e.g., Petersen, Kolen, & Hoover, ; Yen, ) for tracking student growth and progress over the grades. However, these approaches are challenging to develop (e.g., Kolen, ; Patz & Yao, ; Yen, ) and have not been implemented widely (e.g., while the Smarter Balanced tests were based on vertical scales, the other large multistate consortia, PARCC, did not use vertical scales because of several practical limitations; for additional explications on the challenges in implementing vertical scales, see Kannan, ).…”

“…Despite the variety of methods available, in practice, post hoc performance standard articulation methods that employ some variant of impact percentage smoothing (i.e., using a post hoc smoothing procedure to minimize grade‐level differences in percentages of students classified in each performance category) remain extremely popular among the states (see Kannan, , ). This method does not make as many assumptions about the underlying scale and/or the comparability of the tests across the grade levels (see Kannan, ), which explains the popularity of this method, both in empirical evaluations (e.g., Foley, ) and operationally among the 50 states surveyed (see Kannan, ). Nevertheless, the impact percentage smoothing method makes important assumptions about the trajectories for student development that form the basis for the trajectory employed for smoothing (see Kannan, ).…”

“…This method does not make as many assumptions about the underlying scale and/or the comparability of the tests across the grade levels (see Kannan, ), which explains the popularity of this method, both in empirical evaluations (e.g., Foley, ) and operationally among the 50 states surveyed (see Kannan, ). Nevertheless, the impact percentage smoothing method makes important assumptions about the trajectories for student development that form the basis for the trajectory employed for smoothing (see Kannan, ). These assumptions about developmental trajectories are often derived from historical longitudinal data on student performance, but determining such a trajectory ahead of time is not feasible when historical data are unavailable (such as with the next‐generation assessments that were only operationally administered for the first time in Spring 2015).…”

“…The federal mandates have required states to demonstrate growth or progress for students at each of the multiple ALs. However, for results on growth/progress to be reported, at the very least, the performance standards across the grades should be articulated (see Kannan, ). Unfortunately, this was not the case prior to NCLB.…”