Development and Validation of a Vertical Scale for Formative Assessment in Mathematics

Abstract

The regular formative assessment of students' abilities across multiple school grades requires a reliable and valid vertical scale. A vertical scale is a precondition not only for comparing assessment results and measuring progress over time, but also for identifying the most informative items for each individual student within a large item bank independent of the student's grade to increase measurement efficiency. However, the practical implementation of a vertical scale is psychometrically challenging. Several extant studies point to the complex interactions between the practical context in which the scale is used and the scaling decisions that researchers need to make during the development of a vertical scale. As a consequence, clear general recommendations are missing for most scaling decisions. In this study, we described the development of a vertical scale for the formative assessment of third-through ninth-grade students' mathematics abilities based on item response theory methods. We evaluated the content-related validity of this new vertical scale by contrasting the calibration procedure's empirical outcomes (i.e., the item difficulty estimates) with the theoretical, content-related item difficulties reflected by the underlying competence levels of the curriculum, which served as a content framework for developing the scale. Besides analyzing the general match between empirical and content-related item difficulty, we also explored, by means of correlation and multiple regression analyses, whether the match differed for items related to different curriculum cycles (i.e., primary vs. secondary school), domains, or competencies within mathematics. The results showed strong correlations between the empirical and content-related item difficulties, which emphasized the scale's content-related validity. Further analysis showed a higher correlation between empirical and content-related item difficulty at the primary compared with the secondary school level. Across the different curriculum domains and most of the curriculum competencies, we found comparable correlations, implying that the scale is a good indicator of the math ability stated in the curriculum.

Educational gains ofin‐personvs. distance learning in primary and secondary schools: A natural experiment during theCOVID‐19 pandemic school closures in Switzerland

“…All the item parameters needed for this procedure were previously obtained from a much larger sample collected between September 2017 and February 2020 and fixed during WLE estimation. Berger et al (2019) provide a justification of this procedure together with a validation of the item parameters obtained on the curriculum contents for mathematics.…”

Formalizing developmental phenomena as continuous‐time systems: Relations between mathematics and language development

“…In an initial modeling step, data from the years 2018 to 2022 was fit to multiple large unidimensional item response theory‐based models, and individual ability scores for each assessment session were extracted. More details on the Mindsteps software, as well as a discussion of an earlier iteration of the modeling approach, are provided by Tomasik et al (2018) and Berger et al (2019). For pragmatic (i.e., computational) reasons, we subset available data to students with at least 10 assessments (in any of five domains), who have used the system for at least 1 year, and for whom at least four out of five domains have been assessed at least once.…”

Prediction of Performance in Standardised Assessments from Computer-Based Formative Assessment Data

“…As students from different (adjacent) grades may respond to a number of common items, those items that have been presented to students from more than one grade can be used to link the different grades and obtain a vertical scale (Hanson & Béguin, 2002). See Berger et al (2019) for more details about the items and item administration.…”