No abstract

If we wish to embed assessment for accountability within instruction, we need to better understand the relative contribution of different types of learner data to statistical models that predict scores and discrete achievement levels on assessments used for accountability purposes. The present work scales up and extends predictive models of math test scores and achievement levels from existing literature and specifies six categories of models that incorporate information about student prior knowledge, demographics, and performance within the MATHia intelligent tutoring system. Linear regression, ordinal logistic regression, and random forest regression and classification models are learned within each category and generalized over a sample of 23,000+ learners in Grades 6, 7, and 8 over three academic years in Miami-Dade County Public Schools. After briefly exploring hierarchical models of this data, we discuss a variety of technical and practical applications, limitations, and open questions related to this work, especially concerning to the potential use of instructional platforms like MATHia as a replacement for time-consuming standardized tests. Notes for Practice• Advanced educational technologies, including simulations, games, and intelligent tutoring systems, continually assess students in order to provide them with appropriate activities and to determine their mastery of the topics presented.• The assessment embedded in adaptive systems is a type of formative assessment, but we can also use it to make summative conclusions about what a student has learned.• We show that process data collected from students using MATHia, an intelligent tutoring system, over the course of a year can predict high-stakes test scores over and above the ability of a prioryear test to predict these scores.• Models learned on data from a single academic year can be used to predict outcomes for students in other academic years, suggesting that significant predictors of student outcomes remain relatively stable from year to year.• The ability to predict high-stakes exam scores is a necessary (though insufficient) step towards replacing such exams with embedded formative assessments, but even if high-stakes exams remain in place, predictive tools can provide important information about learner readiness for such highstakes exams.

Solving mathematics story problems requires text comprehension skills. However, previous studies have found few connections between traditional measures of text readability and performance on story problems. We hypothesized that recently developed measures of readability and topic incidence measured by text-mining tools may illuminate associations between text difficulty and problem-solving measures. We used data from 3,216 middle and high school students from 10 schools using the Cognitive Tutor Algebra program; these schools were geographically, socioeconomically, racially, and ethnically diverse. We found that several indicators of the readability and topic of story problems were associated with students' tendency to give correct answers and request hints in Cognitive Tutor. We further examined the individual skill of writing an algebraic expression from a story scenario, and examined students at the lowest performing schools in the sample only, and found additional associations for these subsets. Key readability and topic categories that were related to problem-solving measures included word difficulty, text length, pronoun use, sentence similarity, and topic familiarity. These findings are discussed in the context of models of mathematics story problem solving and previous research on text comprehension.

Most authoring tools for intelligent tutoring systems are targeted towards a broad range of applications. Such systems have expressive power but gain the complexity inherent in any general programming language. This paper considers a different kind of authoring tool, focused on creating content for a specific intelligent tutoring system. The resulting system, called pSAT, addresses the great demand for continuing development of content. A system like pSAT needs to be easily learned by end-users and needs to provide feedback adequate for the user to be able to determine that the system will correctly present the content under a wide range of user strategies, preferences and abilities. We focus on design principles driven by these considerations and conclude with a development strategy that begins with a closely-focused content authoring system and then broadens to a system that can more fundamentally affect the type of content presented by the intelligent tutoring system.

No abstract