Automatic plagiarism detection for spoken responses in an assessment of English language proficiency

Wang, Xinhao; Evanini, Keelan; Bruno, James V.; Mulholland, M. Robert

doi:10.1109/slt.2016.7846254

Cited by 5 publications

(8 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Previous work (Wang et al, 2016;Evanini and Wang, 2014) has demonstrated the effectiveness of using content-based features for the task of automatic plagiarized spoken response detection. Therefore, this study investigates the use of improved features based on the measurement of text- Table 1: A sample source passage and the transcription of a sample plagiarized spoken response that was apparently drawn from the source.…”

Section: Content Similaritymentioning

confidence: 99%

“…Features based on the BLEU metric have been proven to be effective in measuring the content appropriateness of spoken responses in the context of English proficiency assessment (Zechner and Wang, 2013) and in measuring content similarity in the detection of plagiarized spoken responses (Wang et al, 2016;Evanini and Wang, 2014). In this study, we first design a new type of feature, known as n-gram overlap, by simulating and improving the previous BLEU-based features.…”

Section: N -Gram Overlapmentioning

confidence: 99%

“…If a test taker uses unseen source materials as the basis for a plagiarized response, the system may fail to detect it. Accordingly, a set of features that do not rely on a comparison with source materials has been proposed previously (Wang et al, 2016). The current study also examined this type of features.…”

Section: Difference In Speaking Proficiencymentioning

confidence: 99%

See 2 more Smart Citations

Application of an Automatic Plagiarism Detection System in a Large-scale Assessment of English Speaking Proficiency

Wang

Evanini

Mulholland

et al. 2019

Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications

Self Cite

View full text Add to dashboard Cite

This study aims to build an automatic system for the detection of plagiarized spoken responses in the context of an assessment of English speaking proficiency for non-native speakers. Classification models were trained to distinguish between plagiarized and nonplagiarized responses with two different types of features: text-to-text content similarity measures, which are commonly used in the task of plagiarism detection for written documents, and speaking proficiency measures, which were specifically designed for spontaneous speech and extracted using an automated speech scoring system. The experiments were first conducted on a large data set drawn from an operational English proficiency assessment across multiple years, and the best classifier on this heavily imbalanced data set resulted in an F1-score of 0.761 on the plagiarized class. This system was then validated on operational responses collected from a single administration of the assessment and achieved a recall of 0.897. The results indicate that the proposed system can potentially be used to improve the validity of both human and automated assessment of non-native spoken English.

show abstract

Section: Content Similaritymentioning

confidence: 99%

Section: N -Gram Overlapmentioning

confidence: 99%

See 1 more Smart Citation

Application of an Automatic Plagiarism Detection System in a Large-scale Assessment of English Speaking Proficiency

Wang

Evanini

Mulholland

et al. 2019

Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications

Self Cite

View full text Add to dashboard Cite

show abstract

“…Potthast et al (2014) give an overview of approaches to detecting plagiarism in written texts based on the systems that competed in the PAN-2014 shared task on plagiarism detection. Wang et al (2016) developed a spoken canned response detection system using similar techniques applied in essay plagiarism detection.…”

Section: Input Capture Filtering Modelsmentioning

confidence: 99%

Atypical Inputs in Educational Applications

Yoon¹,

Cahill²,

Loukina³

et al. 2018

Proceedings of the 2018 Conference of the North American Chapter Of the Association for Computational Linguistics: Hu

View full text Add to dashboard Cite

In large-scale educational assessments, the use of automated scoring has recently become quite common. While the majority of student responses can be processed and scored without difficulty, there are a small number of responses that have atypical characteristics that make it difficult for an automated scoring system to assign a correct score. We describe a pipeline that detects and processes these kinds of responses at run-time. We present the most frequent kinds of what are called non-scorable responses along with effective filtering models based on various NLP and speech processing technologies. We give an overview of two operational automated scoring systems -one for essay scoring and one for speech scoringand describe the filtering models they use. Finally, we present an evaluation and analysis of filtering models used for spoken responses in an assessment of language proficiency.

show abstract

“…Even state-of-the art automated scoring models face challenges in scoring these atypical responses [1,2,3], and researchers in the automated scoring field have tried to solve this issue using a two-step approach where an automated filtering model, as a sub-module of an automated scoring system, filters out atypical responses, and only the remaining responses are scored by the scoring model. A spoken canned response detection system in [4], an off-topic response detection system in [5,6] and a coherence model in [7] are examples of this approach.…”

Section: Introductionmentioning

confidence: 99%

Development of Robust Automated Scoring Models Using Adversarial Input for Oral Proficiency Assessment

et al. 2019

Self Cite

View full text Add to dashboard Cite

In this study, we developed an automated scoring model for an oral proficiency test eliciting spontaneous speech from nonnative speakers of English. In a large-scale oral proficiency test, a small number of responses may have atypical characteristics that make it difficult even for state-of-the-art automated scoring models to assign fair scores. The oral proficiency test in this study consisted of questions asking about content in materials provided to the test takers, and the atypical responses frequently had serious content abnormalities. In order to develop an automated scoring system that is robust to these atypical responses, we first developed a set of content features to capture content abnormalities. Next, we trained scoring models using the augmented training dataset, including synthetic atypical responses. Compared to the baseline scoring model, the new model showed comparable performance in scoring normal responses, while it assigned fairer scores for authentic atypical responses extracted from operational test administrations.

show abstract

Automatic plagiarism detection for spoken responses in an assessment of English language proficiency

Cited by 5 publications

References 13 publications

Application of an Automatic Plagiarism Detection System in a Large-scale Assessment of English Speaking Proficiency

Application of an Automatic Plagiarism Detection System in a Large-scale Assessment of English Speaking Proficiency

Atypical Inputs in Educational Applications

Development of Robust Automated Scoring Models Using Adversarial Input for Oral Proficiency Assessment

Contact Info

Product

Resources

About