A Large Scale Quantitative Exploration of Modeling Strategies for Content Scoring

Abstract

We explore various supervised learning strategies for automated scoring of content knowledge for a large corpus of 130 different content-based questions spanning four subject areas (Science, Math, English Language Arts, and Social Studies) and containing over 230,000 responses scored by human raters. Based on our analyses, we provide specific recommendations for content scoring. These are based on patterns observed across multiple questions and assessments and are, therefore, likely to generalize to other scenarios and prove useful to the community as automated content scoring becomes more popular in schools and classrooms.

Automated text scoring and real‐time adjustable feedback: Supporting revision of scientific arguments involving uncertainty

“…In the case of HASbot, one such gaming behavior would have been students somehow finding a way to get high scores on their explanations and uncertainty attributions without engaging in the type of thinking needed to achieve high scores. HASbot's automated scoring models were not based on the absence or presence of a few keywords (Madnani et al, ). Rather, they were designed to recognize the content of the text through statistically identifiable syntactic mathematized relationships (Heilman & Madnani, ).…”

“…HASbot's automated scoring models were not based on the absence or presence of a few keywords (Madnani et al, 2017). Rather, they were designed to recognize the content of the text through statistically identifiable syntactic mathematized relationships (Heilman & Madnani, 2013b).…”

“…Two approaches are available in automated text scoring. One approach is reference‐based “where responses are scored on the basis of their similarity to reference answers provided by the authors of the question or selected from existing high‐scoring responses” (Madnani, Loukina, & Cahill, , p. 457). In the reference‐based approach, researchers need to conduct a thorough analysis of the content targeted by the question to identify a series of concepts and misconceptions that might be elicited by the question.…”

“…The main function of c‐rater‐ML is to recognize content in open‐ended responses typically written in 300 characters (approximately 50 words) in a specific subject area such as math or science. Spelling mistakes or grammatical errors play a minimal role in automated scoring as long as student responses include the desired content such as scientific principles or data trends (Madnani et al, ). From students’ open‐ended responses, c‐rater‐ML extracts several types of textual features Character n ‐grams: two‐ to five‐character sequences; Word n ‐grams: single‐ and two‐word sequences; Syntax: syntactic relationships between words; Response length bins: rounded logarithm of the total number of characters.…”

“…The main function of c-rater-ML is to recognize content in open-ended responses typically written in 300 characters (approximately 50 words) in a specific subject area such as math or science. Spelling mistakes or grammatical errors play a minimal role in automated scoring as long as student responses include the desired content such as scientific principles or data trends (Madnani et al, 2017) | 603…”