Cost analysis and cost-effectiveness of hand-scored and automated approaches to writing screening

Implications of bias in automated writing quality scores for fair and equitable assessment decisions.

“…ReaderBench uses advanced natural language processing to measure hundreds of indices ranging from simple (e.g., word and sentence length, and unique words used) to sophisticated measures of text characteristics (e.g., lexical chains and discourse connectives). writeAlizer scores generated from ReaderBench output showed validity and reliability coefficients comparable to human ratings of writing quality for timed, narrative samples (Matta, Keller-Margulis, et al, 2022).…”

“…These approaches have been implemented in standardized assessments (e.g., Wechsler Individual Achievement Test [WIAT 4] Essay Composition; Pearson, 2020), for instance, for the identification of school-aged students with writing difficulties as well as graduate school admission (e.g., writing essay of the graduate record examination [GRE]; O’Leary et al, 2018). This technology is attractive given that it reduces time and costs associated with personnel training and scoring procedures (Matta, Keller-Margulis, et al, 2022).…”

Social consequences of educational measures: A commentary.

“…Messick (1995) and AERA (2014) discuss construct-irrelevant easiness and difficulty as “a major source of bias in test scoring and interpretation and of unfairness in test use” (Messick, 1995, p. 743). Matta et al (2023) note that, in scoring writing samples, questions intended to assess writing may depend on knowledge of the content. Students taught the topic in the writing prompt may benefit from construct-irrelevant easiness, whereas those unfamiliar may suffer from construct-irrelevant difficulty.…”

“…Students taught the topic in the writing prompt may benefit from construct-irrelevant easiness, whereas those unfamiliar may suffer from construct-irrelevant difficulty. Students also may misunderstand certain phrases in a test, such as the controlling idea for a paragraph (Matta et al, 2023), if their teachers taught the concept with different words (e.g., central idea , topic sentence ). Powell (2012) reports that many math curricula teach students a misrule: The equals sign suggests an operation with the answer always on the right.…”

“…VanDerHeyden and Solomon (2023) survey the history and rationale behind curriculum-based measurement (CBM) and compare the use and interpretation of fluency and accuracy scoring metrics across multiple constructs (e.g., reading and mathematics). Matta et al (2023) investigate the potential for construct-irrelevant variance to produce differential prediction among racial or ethnic groups using writeAlizer , a software package that evaluates writing samples.…”