Construction and Examination of Math Subskill Mastery Measures

Abstract

This study details the construction of parameters for generating subskill mastery math measures to be used for screening, intervention planning, progress monitoring, and proximal program evaluation. Parameters for generating assessment measures were built and tested to verify initial equivalence of generated measures using potential digits correct as a proxy for task difficulty across generated measures. Generated measures met initial equivalence criteria and were subjected to further reliability analysis. Measures were generated and administered 1 week apart at fall and winter to students in Grades K, 1, 3, 5, and 7. Thirty-four screening measures were examined for delayed alternate form reliability, risk decision agreement, and interobserver agreement. Delayed alternate form reliability values generally exceeded r = .80, could be reliably scored, and yielded consistent risk decisions. Future research directions were discussed.

Mastery measurement in mathematics and the goldilocks effect.

“…These results are comparable to prior studies that focused on more restricted grade ranges, such as Methe et al (2008). Yet, studies of reliability such as VanDerHeyden and Broussard (2020), which employed a common estimator of measurement stability, are limited in that they only allow for the estimation of score stability caused by individual facets of measurement, in this case consistency across two parallel forms (temporal reliability; test–retest reliability), item consistency, and scoring agreement. Such studies do not examine the extent to which scores may be affected by the concurrent influence of each of these facets and the interactions between them.…”

“…In addition, current measures were constructed using a novel process which ensured equality on the maximum score and which randomly generated and then stratified problems, and this process is not widely available. However, given the convergence of results across independent studies (i.e., Hintze, Callahan, et al, 2002; Solomon et al, 2020; VanDerHeyden & Broussard, 2020), we believe that results are applicable to thoughtfully constructed mastery measures. Finally, included students were tested at a timepoint where scores across measures were thought to be more normally distributed.…”

“…Emerging data suggest that mastery measures, which briefly sample (e.g., 2 min) computational fluency across generally equivalent problems umbrellaed under a narrowly conceptualized skill (e.g., 1 × 1 addition), can yield highly reliable scores (Hintze, Christ, et al, 2002; Solomon et al, 2020; VanDerHeyden & Broussard, 2020). As one example of this research, VanDerHeyden and Broussard (2020) investigated delayed alternate form reliability of scores on generated mastery measures in Grades K, 1, 3, 5, and 7 using fall and winter screening measures from the commercial tool sampled presently ( SpringMath ; Education Research & Consulting, Inc, 2013). They reported strong delayed alternate form reliability for fall and winter screening (Median r = .83).…”

Classification Agreement for Gated Screening in Mathematics: Subskill Mastery Measurement and Classwide Intervention

“…Screening measures. Measures were generated according to programmed parameters that have been tested for stability with generated alternate form composite scores ranging from r = .77 to r = .85 in fall and r = .80 to r = .88 in winter (VanDerHeyden & Broussard, 2021). Kindergarten screening measures used in this study have reported alternate form reliability correlations from r = .7 to .84, concurrent correlation validity evidence of r = .44 to .61 with the (Comprehensive Inventory of Basic Skills, Revised Brigance, 1999;VanDerHeyden, Witt, Naquin, & Noell, 2001), r = .49 with the Test of Early Mathematics Ability (Ginsburg & Baroody, 2003), and predictive validity with year-end first grade measures of addition and subtraction of r = .43 (VanDerHeyden et al, 2011).…”

Precision of Single-Skill Mathematics CBM: Group Versus Individual Administration

“…Findings in Stage 1 indicate M-CBMs that evaluate more complex skills, such as problem-solving and multiplication and division computation, tend to be less reliable than measures that sample from less complex skills. More recent research has demonstrated acceptable ( r = .80 or above) alternate forms reliability of generated SSM M-CBMs across elementary grade levels (VanDerHeyden & Broussard, 2021). Contemporary research suggests that M-CBM probes adequately represent the domain of mathematics computation and provide a reliable source of data to make critical and individualized academic decisions for at-risk students (Methe et al, 2015).…”