Summary Street®: Computer Support for Comprehension and Writing

Abstract

Having students express their understanding of difficult, new material in their own words is an effective method to deepen their comprehension and learning. Summary Street® is a computer tutor that offers a supportive context for students to practice this activity by means of summary writing, guiding them through successive cycles of revising with feedback on the content of their writing. Automatic evaluation of the content of student summaries is enabled by Latent Semantic Analysis (LSA). This article describes an experimental study of the comprehension and writing tutor, in which 8th-grade students practiced summary writing over a 4-week period, either with or without the guidance of the tutor. Students using Summary Street® scored significantly higher on an independent comprehension test than the control group for test items that tapped gist level comprehension. Their summaries were also judged to be significantly superior in blind scoring on several measures of writing quality. Students of low-to-moderate achievement levels benefitted most from the tool.

Tutoring Adolescents in Literacy: A meta-analysis

“…Although Lipsey and Wilson (2001) warned against using different types of effect size statistics in the same comparison, we did not want to further reduce the already small number of studies included for the meta-analysis by excluding these studies; we decided that the advantages of including these three conceptually meaningful studies outweighed the risks of excluding them. Furthermore, pretest differences among comparison groups were corrected for three studies (Franzke, Kintsch, Caccamise, Johnson, & Dooley, 2005;Hough et al, 2007;Penney, 2002), whereas other studies did not provide such information.…”

“…Based on the descriptions of the studies, we could identify with certainty only five studies (Allen & Chavkin, 2004;Fisher, 2001;Franzke et al, 2005;Jacobson et al, 2001;Vogelwiesche, Grob, & Winkler, 2006) that included L2 learners. None actually compared groups of L1 and L2 learners.…”

Computer-based assessment of student-constructed responses

“…Teachers and school districts will want to choose materials that map onto mandated curricula or reflect topics covered on high-stakes tests. Some systems, such as iSTART and Summary Street (Franzke et al, 2005) have been developed with this constraint in mind, and the algorithms for analyzing student responses were created such that they did not need extensive engineering to implement with new texts. This can be achieved by automatically identifying expectations based on features of the text (e.g., the current sentence being read is a semantic benchmark for detecting paraphrasing of the current sentence, whereas prior sentences are a benchmark for detecting bridging inferences), but this still requires developing algorithms that can generalize to new texts (Jackson et al, 2010).…”

“…We define student-constructed responses as those that require a student to produce an answer in natural language that may range from a couple of sentences to several paragraphs. These advances have been in the context of computer-based assessments of explanations and think-aloud protocols during reading comprehension (Gilliam, Magliano, Millis, Levinstein, & Boonthum, 2007;Magliano, Millis, the RSAT Development Team, Levinstein, & Boonthum, 2011), the grading of essays and text summaries (Attali & Burstein, 2006;Burstein, Marcu, & Knight, 2003;Franzke, Kintsch, Caccamise, Johnson, & Dooley, 2005;Landauer, Laham, & Foltz, 2003), the grading of short-answer questions (Leacock & Chodorow, 2003), and intelligent tutoring systems and trainers that require students to produce constructed responses during interactive conversations (Graesser, Jeon, & Dufty, 2008;Litman et al, 2006;McNamara, Levinstein, & Boonthum, 2004;VanLehn et al, 2007). These can take the form of directed responses to specific questions or less directed thinkaloud and self-explanation responses.…”

The Effect of Using Automated Essay Evaluation on ESL Undergraduate Students’ Writing Skill

“…A number of AEE systems have sought to provide both automated individualized scores and feedback. Research into AEE varies in scope and includes studies that have explored the usefulness of AEE techniques on writing proficiency (e.g., Rudner & Liang, 2002;Attali, 2004;Franzke et al, 2005;Wang, 2011). And those that have compared human to automated evaluation in terms of reliability and validity (e.g., Cohen et al, 2003;Kulik, 2003;Wang & Brown, 2007;Bejar, 2011;Bejar, 2012).…”