Abstract:E-rater® has been used by the Educational Testing Service for automated essay scoring since 1999. This paper describes a new version of e-rater (V.2) that is different from other automated essay scoring systems in several important respects. The main innovations of e-rater V.2 are a small, intuitive, and meaningful set of features used for scoring; a single scoring model and standards can be used across all prompts of an assessment; modeling procedures that are transparent and flexible, and can be based entirely on expert judgment. The paper describes this new system and presents evidence on the validity and reliability of its scores.
A utomated essay-scoring technologies can enhance both large-scale assessment and classroom instruction. Essay evaluation software not only numerically rates essays but also analyzes grammar, usage, mechanics, and discourse structure. 1,2 In the classroom, such applications can supplement traditional instruction by giving students automated feedback that helps them revise their work and ultimately improve their writing skills. These applications also address educational researchers' interest in individualized instruction. Specifically, feedback that refers explicitly to students' own writing is more effective than general feedback. 3 Our discourse analysis software, which is embedded in Criterion (www.etstechnologies.com), an online essay evaluation application, uses machine learning to identify discourse elements in student essays. The system makes decisions that exemplify how teachers perform this task. For instance, when grading student essays, teachers comment on the discourse structure. Teachers might explicitly state that the essay lacks a thesis statement or that an essay's single main idea has insufficient support. Training the systems to model this behavior requires human judges to annotate a data sample of student essays. The annotation schema reflects the highly structured discourse of genres such as persuasive writing.Our discourse analysis system uses a voting algorithm that takes into account the discourse labeling decisions of three independent systems. The three systems employ natural language processing methods to extract essay-based features that help predict the discourse labels. They also use machine learning to classify the sentences in an essay as particular discourse elements. Our tool automatically labels discourse elements in student essays written on any topic and across writing genres.
Essay-based discourseResearchers have proposed a variety of discourse analysis schemes to capture the semantics of multisentence texts. Some schemes associate a hierarchical representation to a given text, while others a linear one. The representation used in our work is linear. It assumes that essays can be segmented into sequences of discourse spans and that each span is associated with an overall communicative goal. We focus on essay-specific communicative goals, which we encode using intuitive labels that are frequently used in teaching writing, such as thesis statements, main ideas, and conclusion statements.
Essay annotation protocolTo facilitate development of our discourse analysis systems, two human judges annotated several hundred essays. The judges labeled elements in the essay data according to a protocol that explained how to annotate several discourse categories:• Title segments indicate essay titles.• Introductory material segments provide the context or set the stage in which the thesis, a main idea, or the conclusion is to be interpreted. • Thesis segments state the writer's position statement and are related to the essay prompt.• Main idea segments assert the author's main message in co...
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.