Evaluating Procedures for Reducing Measurement Error in Math Curriculum-Based Measurement Probes

Abstract

At present, it is unclear whether math curriculum-based measurement (M-CBM) procedures provide a dependable measure of student progress in math computation because support for its technical properties is based largely upon a body of correlational research. Recent investigations into the dependability of M-CBM scores have found that evaluating probe design procedures is necessary to estimate the contribution of probe-related factors to variance in M-CBM scores. To extend this research, the present study compared a set of commercially available probes (i.e., AIMSweb) with an experimental set that was constructed to enhance consistency in item content across alternative forms. Two probes were randomly selected from each set and administered to 43 students in fifth grade. Generalizability analyses indicated that the experimental probes generated scores with less error variance than those of the AIMSweb probes. Follow-up analyses indicated that fewer administrations of the experimental probes were necessary to attain adequate reliability for relative decisions (i.e., >.80) and high reliability for absolute decisions (i.e., >.90). Results support prior studies and suggest that error variance is associated with M-CBM probe content and development procedures.

Precision of Single-Skill Math CBM Time-Series Data: The Effect of Probe Stratification and Set Size

“…Results of the current study demonstrated that slope estimates were more precise for the stratified condition compared with the randomized condition. Presently, a stratified item resulted in a 12% to 21% decrease in residual error relative to randomized problem assignment depending on the duration of monitoring, which is similar to the reduction in error variance found in Christ and Vining (2006) and Methe et al (2015) for multiple-skill M-CBMs. Although inferior relative to stratified probes, randomized probes still exhibited satisfactory properties and are acceptable for research and practice.…”

“…Christ and Vining (2006) mentioned the possibility of using a stratified item arrangement for SSM-CBM probes but suggested that a random item arrangement is likely sufficient due to the narrower content that is represented on such probes. However, others have expressed concerns related to the precision of SSM-CBM time-series data for instructional decision making (Methe et al, 2015; Solomon et al, in press). Stratifying within SSMs could potentially be a feasible way to increase precision on these probes by increasing the consistency of difficulty across their various iterations.…”

“…We examined two easily manipulated features of CBM-SSM probe construction that extant research suggests may influence precision: item arrangement and set size. Christ and Vining (2006) and Methe et al (2015) investigated the use of a stratified item arrangement relative to randomized item arrangements when using mixed skill M-CBM probes to estimate grade-level math ability. In Christ and Vining (2006), structured probes were created where each column consisted of the same type of math problem and, therefore, each row included one of each type of problem, as defined by each column.…”

Determining Growth Sensitivity of Star Math With a Latent Growth Curve Model

“…To date, research for CAT of achievement has focused largely on Stage 1. Most of the previous research in mathematics progress monitoring measures has been correlational research relying on static data (e.g., Methe et al, 2015), confirming that STAR-M meet the standards of technical adequacy at Stage 1. The following section focuses on Stage 2 research related to progress monitoring measures.…”

Development of curriculum-based measurements in mathematical computations for Arab-speaking fourth grade students

“…Although MC-CBM has gained robust empirical support, the findings concerning the psychometric properties of the MC-CBM used metric (i.e., digits correct per unit of time) and the assessment context (single-skill or multiple-skill) are inconclusive (Christ et al., 2005; Hintze et al., 2002; Methe, Briesch, and Hulac, 2015; Strait, Smith, Pender, Malone, Roberts, & Hall, 2015). Christ et al.…”