Using Instruction-Embedded Formative Assessment to Predict State Summative Test Scores and Achievement Levels in Mathematics

Abstract

If we wish to embed assessment for accountability within instruction, we need to better understand the relative contribution of different types of learner data to statistical models that predict scores and discrete achievement levels on assessments used for accountability purposes. The present work scales up and extends predictive models of math test scores and achievement levels from existing literature and specifies six categories of models that incorporate information about student prior knowledge, demographics, and performance within the MATHia intelligent tutoring system. Linear regression, ordinal logistic regression, and random forest regression and classification models are learned within each category and generalized over a sample of 23,000+ learners in Grades 6, 7, and 8 over three academic years in Miami-Dade County Public Schools. After briefly exploring hierarchical models of this data, we discuss a variety of technical and practical applications, limitations, and open questions related to this work, especially concerning to the potential use of instructional platforms like MATHia as a replacement for time-consuming standardized tests. Notes for Practice• Advanced educational technologies, including simulations, games, and intelligent tutoring systems, continually assess students in order to provide them with appropriate activities and to determine their mastery of the topics presented.• The assessment embedded in adaptive systems is a type of formative assessment, but we can also use it to make summative conclusions about what a student has learned.• We show that process data collected from students using MATHia, an intelligent tutoring system, over the course of a year can predict high-stakes test scores over and above the ability of a prioryear test to predict these scores.• Models learned on data from a single academic year can be used to predict outcomes for students in other academic years, suggesting that significant predictors of student outcomes remain relatively stable from year to year.• The ability to predict high-stakes exam scores is a necessary (though insufficient) step towards replacing such exams with embedded formative assessments, but even if high-stakes exams remain in place, predictive tools can provide important information about learner readiness for such highstakes exams.

Self-Made Video Tutorials in Mathematics 8

“…Summative evaluations are often administered during a designated "test time" with minimal interruptions and student engagement. According to Zheng et al, (2019), this "test time" is separate from instructional time. Indeed, interpreting summative tests assumes no learning occurs during the exam, and the atmosphere is typically constructed to support that idea.…”

Do Written Responses to Open-Ended Questions on Fourthgrade Online Formative Assessments in Mathematics Help Predict Scores on End-of-Year Standardized Tests?

“…From a sample of 23,000 learners in Grades 6, 7, and 8 over three academic years, Zheng, G. (2019) analyzes the relative contribution of different types of learner data to statistical models that predict scores on summative assessments. They use six different categories of statistical models to predict summative scores.…”

Motivational and Emotional Orientation, Engagement, and Achievement in Mathematics. A Case Study With One Sixth-Grade Classroom Working With an Electronic Textbook on Fractions

“…They can rather easily be administered by "counting how many actions students take within the system" (Hoch et al, 2018b, 843). They can encompass, for instance, the number of postings in an online forum (Henrie et al, 2015), the number of completed tasks in a learning environment (Hew et al, 2016;Huang et al, 2019), or how often students ask for assistance autonomously (Hoch, 2021;Feng et al, 2006;Zheng et al, 2019). Although research suggests a tendency of more frequent task behavior (i.e., higher scores in such count measures) being positively related to higher learning outcomes (Henrie et al, 2015;Junco and Clem, 2015;Hew et al, 2016;Huang et al, 2019), such relation-again-may not be that simple or direct as anticipated, as individual characteristics, such as prior knowledge, may moderate this effect (Hoch, 2021).…”

“…In particular, engagement with interactive exercises is indicated by two count measures, task count (the total amount of tasks solved by each student as in Feng et al, 2006;Huang et al, 2019) and exercise count (the number of different exercises each student accessed as in Zheng et al, 2019), as well as a time measure, the total problem solving time (for all tasks, e.g., Feng et al, 2006;Zheng et al, 2019). Usage of the material's hint system is measured by hint count (the total number of requested hints, e.g., Anozie and Junker, 2006;Feng et al, 2006;Zheng et al, 2019). How students made use of the offered adaptive feedback is represented by their feedback time (mean time in feedback phases, e.g., Anozie and Junker, 2006;Ayers and Junker, 2008).…”