Off-Topic Essay Detection: A Systematic Review

Abstract

Abstract. Essays are widely used for learning assessment in the educational

Off-Topic Essay Detection: A comparative study on the Portuguese language

“…This study addresses the detection of essays from unexpected prompt, which can be seen as a task of analyzing the textual similarity between the essay and the prompt statement (Higgins et al, 2006). In the existing literature, off-topic essay detection has been performed by applying techniques of natural language processing, semantic analysis and linguistics features, such as essay length, organization, and sentence variety (Passero et al, 2017).…”

“…A previous systematic review of the literature identified five articles, published between 2006 and 2016, which present one or more approaches for off-topic essay detection (Passero et al, 2017). In this comparative study, these approaches were considered to represent the state of the art, together with the approaches presented by Klebanov, Flor, & Gyawali (2016) and Rei & Cummins (2016), which were found to be relevant and adaptable to the task of off-topic essay detection.…”

“…Table 1, adapted from the results of Passero et al (2017), describes the existing approaches for off-topic essay detection that were included in this comparative study. The study of Chen & Zhang (2016), although described in Passero et al (2017), was not included in Table 1 because it did not present a novel approach. 180…”

“…The performance of the algorithms was measured by the overall rate of correct classifications (accuracy), precision, recall, false positive rate and false negative rate. These measures were used because they are present in the literature related to off-topic essay detection, as presented by Passero et al (2017), and, therefore, aim to facilitate the comparison of the results achieved with the one found in other studies.…”

“…Such essay may receive a zero score in high-stakes tests. The current state of art of automated off-topic essay detection shows some deficiencies, including: the low accuracy of current tested approaches, the use of artificial validation sets, and the lack of studies focused on the Portuguese language (Passero, Ferreira, Haendchen Filho, & Dazzi, 2017). Thus, it can be seen that the automated detection of off-topic essays is an ongoing task.…”