Estimating the Effect of Web-Based Homework

Abstract

Traditional studies of intelligent tutoring systems have focused on their use in the classroom. Few have explored the advantage of using ITS as a web-based homework (WBH) system, providing correctness-only feedback to students. A second underappreciated aspect of WBH is that teachers can use the data to more efficiently review homework. Universities across the world are employing these WBH systems but there are no known comparisons of this in K12. In this work we randomly assigned 63 thirteen and fourteen year olds to either a traditional homework condition (TH) involving practice without feedback or a WBH condition that added correctness feedback at the end of a problem and the ability to try again. All students used ASSISTments, an ITS, to do their homework but we ablated all of the intelligent tutoring aspects of hints, feedback messages, and mastery learning as appropriate to the two practice conditions. We found that students learned reliably more in the WBH condition with an effect size of 0.56. Additionally, teacher use of the homework data lead to a more robust and systematic review of the homework. While the resulting increase in learning was not significantly different than the TH review, the combination of immediate feedback and teacher use of the data provided by WBH resulted in increased learning compared to traditional homework practices. Future work will further examine modifications to WBH to further improve learning from homework and the role of WBH in formative assessment.

The ASSISTments Ecosystem: Building a Platform that Brings Scientists and Teachers Together for Minimally Invasive Research on Human Learning and Teaching

“…Posttest results show a reliable effect of condition and a meaningful effect size of 0.6. In follow-up studies we have found similar findings that showed ASSISTments mediated immediate feedback lead to better learning than the business as usual next day feedback (Kelly, et al 2013b;Singh et al 2011).…”

“…Students in the control condition were given homework to replicate the traditional scenario of getting feedback the next day, whereas students in the experimental condition received correctness-only feedback as they did their homework. In Kelly et al (2013b), we found a reliable increase in student knowledge when students got immediate feedback on their homework, compared to next-day feedback. The effect size was half a standard deviation.…”

“…The study reported that students learned a great deal from teachers reviewing homework with the data from the reports, Fig. 6 and 7 (Kelly et al 2013b). For example, the teacher used the report in Fig.…”

“…About a dozen universities now are using ASSISTments to run studies of some sort. The tool has been used to do randomized controlled trials reported in over 18 peerreviewed publications (Broderick et al 2012;Heffernan et al 2012a, b;Kehrer et al 2013;Kelly et al 2013a;Kelly et al 2013b;Kim et al 2009;Mendicino et al 2009;Ostrow and Heffernan 2014a, b;Pardos et al 2011;Razzaq et al 2005Razzaq et al , 2009Razzaq et al 2007;Razzaq et al 2008;Razzaq and Heffernan 2006;2009;Sao Pedro et al 2009;Shrestha et al 2009;Singh et al 2011;Walonoski and Heffernan 2006). Those studies were generally about comparing different ways of giving feedback to students, and measuring learning on a posttest.…”

Evaluating Student-Facing Learning Dashboards of Affective States

“…In addition, ITSs typically generate and maintain a student model, which might create some interesting opportunities for dashboards. Exceptions are work by Lovett et al (2008) who report on instructors using reports from an ITS in an online course [10], by Arroyo et al (2014) who describe teacher reports generated by an ITS [4], and by Kelly et al (2013) who study how a teacher used a report from a web-based homework system to decide what parts of the homework to review in class [7].…”