Exploration of New Complexity Metrics for Curriculum-Based Measures of Writing

Abstract

Researchers and practitioners have questioned whether scoring procedures used with curriculum-based measures of writing (CBM-W) capture growth in complexity of writing. We analyzed data from six independent samples to examine two potential scoring metrics for picture word CBM-W (PW), a sentence-level CBM task. Correct word sequences per response (CWSR) and words written per response (WWR) were compared with the current standard metric of correct word sequences (CWS). Linear regression analyses indicated that CWSR predicted scores on standardized norm-referenced criterion measures in more samples than did WWR or CWS. Future studies should explore the capacity of CWSR and WWR to show growth over time, stability, diagnostic accuracy, and utility for instructional decision making.

Aspects of Technical Adequacy of an Early-Writing Measure for English Language Learners in Grades 1 to 3

“…Although the findings here are generally in line with prior research examining CBM-PW with the general population (i.e., technical adequacy is stronger for second grade than for first grade; McMaster et al, 2011), there were a couple of interesting contradictions between this study and prior studies (i.e., Keller-Margulis et al, 2016; McMaster et al, 2011). In contrast to CBM-PW studies with the general population (McMaster et al, 2011; Wagner et al, 2018), metrics incorporating sentence-level features in this study (i.e., CWS, C-IWS, CWSR) did not perform as well or as consistently as word-level metrics (i.e., WW, WSC). This could be a distinct feature of EL writing progression, an anomaly distinct to this study’s sample, or a feature specific to the criterion measure (ACCESS).…”

“…In addition, CBM-PW provides scaffolds in the form of pictures and keywords that may promote access for ELs with beginning levels of ELP. CBM-PW is scored using a variety of metrics that may be roughly categorized as production (i.e., total words written [WW]), accurate-production (i.e., total words spelled correctly [WSC], total correct word sequences [CWS], and correct minus incorrect word sequences [C-IWS]), production-independent (i.e., %WSC and %CWS; McMaster et al, 2014), and complexity (i.e., correct word sequences per response [CWSR]; Wagner et al, 2018). CWS is defined as two adjacent words that are both spelled correctly and used appropriately within the context of the sentence, inclusive of capitalization and punctuation (McMaster et al, 2014).…”

“…CWS is defined as two adjacent words that are both spelled correctly and used appropriately within the context of the sentence, inclusive of capitalization and punctuation (McMaster et al, 2014). CWSR is calculated by dividing the number of attempted sentences, with each sentence considered a response, by CWS to produce an average number of CWS per sentence/response (Wagner et al, 2018).…”

Scoring measures of word dictation curriculum‐based measurement in writing: Effects of incremental administration

“…Future research must continue to examine the validity of word level probes as well as the scoring methods that are commonly used. Research over about the last 15 years has begun to explore a number of alternative scoring methods for sentence and story/paragraph level CBM‐W (e.g., Allen, Poch, & Lembke, ; Gansle, Noell, VanDerHeyden, Naquin, & Slider, ; Wagner et al, ). It is possible that comparable scoring indices such as the mean length of correct letter sequences or the average number of correct letter sequences per response may provide additional information about students’ spelling progress that also contains acceptable technical adequacy.…”

A literature synthesis of curriculum‐based measurement in writing for English learners

“…Of the three components of CAF, complexity is the least examined in CBM‐W research (Wagner et al, 2019). While some production‐dependent scoring procedures may account for aspects of complexity (e.g., C‐IWS), complexity has proven difficult to define and measure.…”