Computer Analysis of the Toefl Test of Written English

Abstract

This project had three main objectives: (1) to establish a database of essays written by different language groups on a variety of topics for the Test of Written English (TWE) that can be used in future research; (2) to summarize, analyze, and compare the linguistic properties of those essays; and (3) to determine how the TWE performance of language groups relates to essay styles. As part of the first objective of this project we created a database of 1,737 essays, a data matrix of essay variables, files containing sorted phrases and vocabularies of different language groups, and files of the common and unique vocabulary items for each pair of language groups.The essay sample consisted of TWE essays from five language groups, including Arabic, Chinese, English (including native-English and nonnative-English), and Spanish speakers. Essays from English-speaking students in the United States were collected and scored to provide a baseline with which to compare essays by students who speak English as a second language (ESL).This report includes the analysis of 106 variables for each essay along with summary analyses, such as correlation, analyses of variance, discriminative analysis, and factor analysis. It also presents information on the accuracy and cost of data entry and on the accuracy of the text analysis programs, as well as extensive data on vocabulary. A program that assesses content was developed for this project.Current (1998-99) members of the TOEFL Committee of Examiners are:

ANALYTIC SCORING OF TOEFL® CBT ESSAYS: SCORES FROM HUMANS AND E‐RATER®

“…This was not completely unexpected, given previous research findings on the strong relationships between essay length and holistic scores (Carson, Bridgeman, Camp, & Waanders, 1985;Ferris, 1994;Frase, Faletti, Ginther, & Grant, 1999;Grant & Ginther, 2000;Jarvis, 2002;Jarvis et al, 2003;Reid, 1986) and between lexical diversity measures and holistic scores (Engber, 1995;Laufer & Nation, 1995). Interestingly, the holistic and development scores were found to be most highly correlated to essay length, while the mechanics score was least correlated to essay length.…”

Analysis of Discourse Features and Verification of Scoring Levels for Independent and Integrated Prototype Written Tasks for the New Toefl®

“…Lexical sophistication, analyzed in two ways: a. average word length (Biber, 1986(Biber, , 1988(Biber, , 1995Chipere, Malvern, Duran, & Richards, 2003;Engber, 1995;Frase, Faletti, Ginther, & Grant, 1999;Grant & Ginther, 2000;Reppen, 1994) Our operational definition (following Engber, 1995) appears in Appendix A. Lexical sophistication, analyzed in two ways: a. average word length (Biber, 1986(Biber, , 1988(Biber, , 1995Chipere, Malvern, Duran, & Richards, 2003;Engber, 1995;Frase, Faletti, Ginther, & Grant, 1999;Grant & Ginther, 2000;Reppen, 1994) Our operational definition (following Engber, 1995) appears in Appendix A.…”

A hierarchical classification approach to automated essay scoring

“…One proxy for the fluency of a writer is the length of the essay. Essays are often distinguished in terms of their length, where essays with higher scores are longer than lower rated essays (e.g., Crossley, Weston, McLain Sullivan, & McNamara, 2011;Ferris, 1994;Frase, Faletti, Ginther, & Grant, 1997;Guo, Crossley, & McNamara, 2013;Jarvis, Grant, Bikowski, & Ferris, 2003;McNamara, Crossley, & McCarthy, 2010;McNamara et al, 2013). Shorter essays have fewer words and fewer paragraphs, which in turn are indicative of less fluency on the part of the writer.…”