Content Importance Models for Scoring Writing From Sources

Klebanov, Beata Beigman; Madnani, Nitin; Burstein, Jill; Somasundaran, Swapna

doi:10.3115/v1/p14-2041

Cited by 19 publications

(14 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One general approach has been to compare the content in the test response with elements from the stimulus materials presented to the test taker in the source‐based task, such as a listening passage or an article. This approach has resulted in some features that have significant correlations with human scores but that do not perform as well as the features calculated using models trained on human‐scored responses; Evanini, Xie, and Zechner () have presented results using this type of prompt‐based feature for spoken responses, and Beigman Klebanov, Madnani, Burstein, and Somasundaran () have presented results using prompt‐based features for essays.…”

Section: Discussionmentioning

confidence: 99%

Automated Scoring of Nonnative Speech Using the SpeechRater^SMv. 5.0 Engine

Chen

Zechner

Yoon

et al. 2018

ETS Research Report Series

View full text Add to dashboard Cite

This research report provides an overview of the R&D efforts at Educational Testing Service related to its capability for automated scoring of nonnative spontaneous speech with the SpeechRaterSM automated scoring service since its initial version was deployed in 2006. While most aspects of this R&D work have been published in various venues in recent years, no comprehensive account of the current state of SpeechRater has been provided since the initial publications following its first operational use in 2006. After a brief review of recent related work by other institutions, we summarize the main features and feature classes that have been developed and introduced into SpeechRater in the past 10 years, including features measuring aspects of pronunciation, prosody, vocabulary, grammar, content, and discourse. Furthermore, new types of filtering models for flagging nonscorable spoken responses are described, as is our new hybrid way of building linear regression scoring models with improved feature selection. Finally, empirical results for SpeechRater 5.0 (operationally deployed in 2016) are provided.

show abstract

Section: Discussionmentioning

confidence: 99%

Automated Scoring of Nonnative Speech Using the SpeechRater^SMv. 5.0 Engine

Chen

Zechner

Yoon

et al. 2018

ETS Research Report Series

View full text Add to dashboard Cite

show abstract

“…We use Galley and McKeown (2003) lexical chaining and extract the first set of features (LEX1) introduced in Somasundaran et al (2014). We do not implement the second set because we do not have the annotation or the tagger to tag discourse cues.…”

Section: Methodsmentioning

confidence: 99%

“…Lexical chains (Somasundaran et al, 2014) and entity grids (Burstein et al, 2010) have been used to measure lexical cohesion. In other words, these models measure the continuity of lexical meaning.…”

Section: Topic-grid and Topic Chainsmentioning

confidence: 99%

“…For example, in Somasundaran et al (2014) the coherence elements are adherence to the essay topic, elaboration, usage of varied vocabulary, and sound organization of thoughts and ideas. In Crossley and McNamara (2011) the elements are effective lead, clear purpose, clear plan, topic sentences, paragraph transitions, organization, unity, perspective, conviction, grammar, syntax, and mechanics.…”

Section: Introductionmentioning

confidence: 99%

“…Entity-based essay representation along with type/token ratios for each syntactic role is another method to evaluate coher-ence (Burstein et al, 2010) that is shown in Burstein et al (2013) to be a predictive model on a corpus of essays from grades 6-12. Lexical chaining addresses multiple aspects of coherence such as elaboration, usage of varied vocabulary, and sound organization of thoughts and ideas (Somasundaran et al, 2014). Discourse structure is used to measure the organization of argumentative writing (Cohen, 1987;Burstein et al, 1998;Burstein et al, 2003b).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications

Tetreault

Burstein

Leacock³

2015

View full text Add to dashboard Cite

iii IntroductionWe are excited to be holding the 10th anniversary the BEA workshop. Since starting in 1997, the BEA workshop, now one of the largest workshops at NAACL/ACL, has become one of the leading venues for publishing innovative work that uses NLP to develop educational applications. The consistent interest in and growth of the workshop has clear ties to societal need and related advances in the technology, and the maturity of the NLP/education field. NLP capabilities now support an array of learning domains, including writing, speaking, reading, and mathematics. Within these domains, the community continues to develop and deploy innovative NLP approaches for use in educational settings. In the writing and speech domains, automated writing evaluation (AWE) and speech scoring applications, respectively, are commercially deployed in high-stakes assessment and instructional settings, including Massive Open Online Courses (MOOCs). We also see widely-used commercial applications for plagiarism detection and peer review. Major advances in speech technology, have made it possible to include speech in both assessment and Intelligent Tutoring Systems. There has been a renewed interest in spoken dialog and multi-modal systems for instruction and assessment as well as feedback. We are also seeing explosive growth of mobile applications for game-based applications for instruction and assessment. The current educational and assessment landscape, continues to foster a strong interest and high demand that pushes the state-of-the-art in AWE capabilities to expand the analysis of written responses to writing genres other than those traditionally found in standardized assessments, especially writing tasks requiring use of sources and argumentative discourse.The use of NLP in educational applications has gained visibility outside of the NLP community. First, the Hewlett Foundation reached out to public and private sectors and sponsored two competitions: one for automated essay scoring, and the other for scoring of short answer, fact-based response items. The motivation driving these competitions was to engage the larger scientific community in this enterprise. MOOCs are now beginning to incorporate AWE systems to manage the thousands of constructedresponse assignments collected during a single MOOC course. Learning@Scale is a recent venue for discussing NLP research in education. The NLP-TEA workshop, now in its second year (NLP-TEA2), gives special attention to papers working on Asian languages. The Speech and Language Technology in Education (SLaTE), now in its sixth year, promotes the use of speech and language technology for educational purposes. Another breakthrough for educational applications within the CL community is the presence of a number of shared-task competitions over the last three years. There have been three shared tasks on grammatical error correction with the most recent edition hosted at CoNLL 2014. In 2014 alone, there were four shared tasks for NLP and Education-related areas.As a community, we contin...

show abstract

An Investigation of the e‐rater^® Automated Scoring Engine's Grammar, Usage, Mechanics, and Style Microfeatures and Their Aggregation Model

Jing

Zhang

Bejar

2017

ETS Research Report Series

View full text Add to dashboard Cite

Automated essay scoring (AES) generally computes essay scores as a function of macrofeatures derived from a set of microfeatures extracted from the text using natural language processing (NLP). In the e‐rater® automated scoring engine, developed at Educational Testing Service (ETS) for the automated scoring of essays, each of four macrofeatures (grammar, usage, mechanics, and style [GUMS]) is computed from a set of microfeatures. Statistical analyses reveal that some of these microfeatures might not explain much of the variance in human scores regardless of the writing tasks. Currently, the microfeatures in the same macrofeature group are equally weighted to produce the macrofeature score. We propose an alternative weighting scheme that gives higher weights to the microfeatures that are more predictive of human scores in each macrofeature group. Our results suggest that even though there is negligible difference between the proposed and the current equal weighting schemes and the current model in terms of the prediction of human scores and the correlation with external measures, our scheme improves the consistency of the resultant macrofeature scores across writing tasks to a considerable extent.

show abstract

Content Importance Models for Scoring Writing From Sources

Cited by 19 publications

References 11 publications

Automated Scoring of Nonnative Speech Using the SpeechRater^SMv. 5.0 Engine

Automated Scoring of Nonnative Speech Using the SpeechRater^SMv. 5.0 Engine

Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications

An Investigation of the e‐rater^® Automated Scoring Engine's Grammar, Usage, Mechanics, and Style Microfeatures and Their Aggregation Model

Contact Info

Product

Resources

About

Content Importance Models for Scoring Writing From Sources

Cited by 19 publications

References 11 publications

Automated Scoring of Nonnative Speech Using the SpeechRaterSMv. 5.0 Engine

Automated Scoring of Nonnative Speech Using the SpeechRaterSMv. 5.0 Engine

Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications

An Investigation of the e‐rater® Automated Scoring Engine's Grammar, Usage, Mechanics, and Style Microfeatures and Their Aggregation Model

Contact Info

Product

Resources

About

Automated Scoring of Nonnative Speech Using the SpeechRater^SMv. 5.0 Engine

Automated Scoring of Nonnative Speech Using the SpeechRater^SMv. 5.0 Engine

An Investigation of the e‐rater^® Automated Scoring Engine's Grammar, Usage, Mechanics, and Style Microfeatures and Their Aggregation Model