Construct Validity of E‐rater® in Scoring Toefl® Essays

ETS is an Equal Opportunity/Affirmative Action Employer.As part of its educational and social mission and in fulfilling the organization's non-profit Charter and Bylaws, ETS has and continues to learn from and also to lead research that furthers educational and measurement research to advance quality and equity in education and assessment for all users of the organization's products and services.Copyright © 2011 by ETS. All rights reserved.No part of this report may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, without permission in writing from the publisher. Violators will be prosecuted in accordance with both U.S. and international copyright laws.CRITERION, E-RATER, ETS, the ETS logos, GRADUATE RECORD EXAMINATIONS, GRE, LISTENING. LEARNING. LEADING., TOEFL, TOEFL iBT, the TOEFL logo, and TWE are registered trademarks of Educational Testing Service (ETS).COLLEGE BOARD and SAT are registered trademarks of the College Entrance Examination Board.i Abstract Automated scoring has the potential to dramatically reduce the time and costs associated with the assessment of complex skills such as writing, but its use must be validated against a variety of criteria for it to be accepted by test users and stakeholders. This study addresses two validityrelated issues regarding the use of e-rater ® with the independent writing task on the TOEFL iBT ® (Internet-based test). First, relationships between automated scores of iBT tasks and nontest indicators of writing ability were examined. This was followed by exploration of prompt-related differences in automated scores of essays written by the same examinees. Correlations between both human and e-rater scores and nontest indicators were moderate but consistent, with few differences between e-rater and human rater scores. E-rater was more consistent across prompts than individual human raters, although there were differences in scores across prompts for the individual features used to generate total e-rater scores. ETS administers the TOEFL program under the general direction of a policy board that was established by, and is affiliated with, the sponsoring organizations. Members of the TOEFL Board (previously the Policy Council) represent the College Board, the GRE Board, and such institutions and agencies as graduate schools of business, two-year colleges, and nonprofit educational exchange agencies.   Since its inception in 1963, the TOEFL has evolved from a paper-based test to a computer-based test and, in 2005, to an Internet-based test, TOEFL iBT ® . One constant throughout this evolution has been a continuing program of research related to the TOEFL test. From 1977 to 2005, nearly 100 research and technical reports on the early versions of TOEFL were published. In 1997, a monograph series that laid the groundwork for the development of TOEFL iBT was launched. With the release of TOEFL iBT, a TOEFL iBT report series has been introduced.Currently this r...

Section: Methodssupporting

confidence: 74%

Section: Implications and Future Directionssupporting

confidence: 59%

Section: Methodsmentioning

confidence: 99%

Section: Task Variability In Writing Assessmentmentioning

confidence: 99%

See 2 more Smart Citations

VALIDATION OF AUTOMATED SCORES OF TOEFL IBT^®TASKS AGAINST NONTEST INDICATORS OF WRITING ABILITY

Weigle

2011

“…First, factor analyses were performed to investigate the structure of e-rater features when the DWU measures are added to the set. Factor analyses of both TOEFL computer-based test essays (Attali, 2007) and essays written by native English speakers from a wide developmental range (4th to 12th grade; Attali & Powers, 2008, 2009) revealed a similar underlying structure of the e-rater features. This three-factor structure has an attractive hierarchical linguistic interpretation with a word choice factor, a grammaticalconventions-within-a-sentence factor, and a fluency factor.…”

Section: Evaluation Setupmentioning

confidence: 93%

A Differential Word Use Measure for Content Analysis in Automated Essay Scoring

Attali

2011

Self Cite

This paper proposes an alternative content measure for essay scoring, based on the difference in the relative frequency of a word in high-scored versus low-scored essays. The differential word use (DWU) measure is the average of these differences across all words in the essay. A positive value indicates the essay is using vocabulary more typical of high-scoring essays in this task, and vice versa. In addition to the traditional prompt level, this measure is also computed at the generic task level, where the content of an essay is evaluated in the context of the general vocabulary of the writing task (e.g., GRE ® issue), across different prompts. Evaluation results across four GRE and TOEFL ® tasks are presented. Factor analyses show that both the prompt and task DWU measures load on the same factor as the prompt-specific content analysis measures of e-rater ® . Regression results on the human essay scores show that the generic task DWU measure is a strong predictor of human scores, second only to essay length among noncontent e-rater features. This measure provides a way to conduct a task-level content analysis that is related to the prompt-specific content analysis of e-rater but does not require promptspecific training. Regression results for the prompt-level DWU show that it can be used as a replacement to prompt-specific content-vector analysis (CVA) features.

Monitoring of Scoring Using the e‐rater^® Automated Scoring System and Human Raters on a Writing Test

Wang

Davier

2014

This article proposes and investigates several methodologies for monitoring the quality of constructed‐response (CR) scoring, both human and automated. There is an increased interest in the operational scoring of essays using both automated scoring and human raters. There is also evidence of rater effects—scoring severity and score inconsistency by human raters. Recently, automated scoring of CRs was successfully implemented with human scoring for operational programs (TOEFL® and GRE® tests); however, there is much that is not yet known about the performance of automated scoring systems. Hence, for quality assurance purposes, there is the need to provide a consistent and standardized approach to monitor the quality of the CR scoring over time and across programs. Monitoring the scoring results will help provide scores that are both fair and accurate for test takers and test users, enabling testing programs to detect and correct changes in the severity of scoring.