Identifying off-topic student essays without topic-specific training data

Abstract

Educational assessment applications, as well as other natural-language interfaces, need some mechanism for validating user responses. If the input provided to the system is infelicitous or uncooperative, the proper response may be to simply reject it, to route it to a bin for special processing, or to ask the user to modify the input. If problematic user input is instead handled as if it were the system's normal input, this may degrade users' confidence in the software, or suggest ways in which they might try to "game" the system. Our specific task in this domain is the identification of student essays which are "off-topic", or not written to the test question topic. Identification of off-topic essays is of great importance for the commercial essay evaluation system Criterion SM . The previous methods used for this task required 200-300 human scored essays for training purposes. However, there are situations in which no essays are available for training, such as when users (teachers) wish to spontaneously write a new topic for their students. For these kinds of cases, we need a system that works reliably without training data. This paper describes an algorithm that detects when a student's essay is off-topic without requiring a set of topic-specific essays for training. This new system is comparable in performance to previous models which require topic-specific essays for training, and provides more detailed information about the way in which an essay diverges from the requested essay topic.

Atypical Inputs in Educational Applications

“…Off-topic responses and generic responses are unrelated to the prompt, while the promptrepetition responses and canned responses can be considered repetition or plagiarism. For automated essay scoring, off-topic detection systems have been developed based on question-specific content models, such as a standard vector space model (VSM) built for each question (Bernstein et al, 2000;Higgins et al, 2006;Louis and Higgins, 2010). …”

“…"Banging on the keyboard" can be identified by analyzing part-ofspeech sequences and looking for ill-formed sequences (Higgins et al, 2006).…”

Off-Topic Essay Detection: A Systematic Review

“…When the prompt-adherence criterion is not met, an essay may be rated as off-topic. Off-topic essays can be regarded as of two major types [Higgins, Burstein and Attali 2006]: -Unexpected Topic: possibly well-written essays that do not address the expected topic; -Bad-faith: essays that mainly consist of text copied from the prompt or with irrelevant musings, such as purposely inserted chunks of text unrelated to the topic and the essay itself.…”

“…The detection of off-topic essays can be seen as a task of analyzing the closeness between the content of an essay and the prompt statement [Higgins, Burstein and Attali 2006]. Linguistic features such as essay length, organization, and sentence variety are also relevant for off-topic essay detection [Chen and Zhang 2016].…”

“…One solution to mitigate this problem is to use an off-topic essay detection tool alongside or embedded into the existing essay scoring systems Heilman 2014, Chen andZhang 2016]. A tool for automatic off-topic essay detection might also be used to generate relevant feedback to students [Higgins, Burstein and Attali 2006].…”