Harnessing the potential of trace data and linguistic analysis to predict learner performance in a multi‐text writing task

Abstract

Background: Assignments that involve writing based on several texts are challenging to many learners. Formative feedback supporting learners in these tasks should be informed by the characteristics of evolving written product and by the characteristics of learning processes learners enacted while developing the product. However, formative feedback in writing tasks based on multiple texts has almost exclusively focused on essay product and rarely included SRL processes.Objectives: We explored the viability of using product and process features to develop machine learning classifiers that identify low-and high-performing essays in a multi-text writing task.Methods: We examined learning processes and essay submissions of 163 graduate students working on an authentic multi-text writing assignment. We utilised learners' trace data to obtain process features and state-of-the-art natural language processing methods to obtain product features for our classifiers. Results and Conclusions: Of four popular classifiers examined in this study, RandomForest achieved the best performance (accuracy = 0.80 and recall = 0.77). The analysis of important features identified in the Random Forest classification model revealed one product (coverage of reading topics) and three process (elaboration/organisation, rereading and planning) features as important predictors of writing quality.Major Takeaways: The classifier can be used as a part of a future automated writing evaluation system that will support at scale formative assessment in writing tasks based on multiple texts in different courses. Based on important predictors of essay performance, a guidance can be tailored to learners at the outset of a multi-text writing task to help them do well in the task.

Predictors of middle school students’ perceptions of automated writing evaluation

“…Based on students' open-ended survey responses, one possibility might be that more proficient writers did not receive the feedback they needed from MI Write, perhaps due to noted limitations in AWE feedback on higher-level writing skills ( Deane, 2013 ). Researchers and AWE developers should continue to expand the assessment and feedback capabilities of their systems to provide feedback on higher-level writing skills (e.g., Correnti et al, 2022 ; Raković et al, 2021 ), as well as aspects of the writer's process, not just the writing product (see Deane et al, 2021 ; Raković et al, 2022 ).…”

Artificial intelligence on the advance to enhance educational assessment: Scientific clickbait or genuine gamechanger?

“…While Flor and Andrews‐Todd (2022) built separate classifiers from log and text data, Raković et al (2023) built one classifier based on log and text data from a writing task based on multiple texts. The classifier is a set of process indicators of self‐regulated learning and linguistic product features of the learners' essays.…”

Text Linguistics and the Use of Linguistic Data in Modern Technologies: Prospects for Development

“…It can be simple or complex, composed of several syntactic clauses. The linguists Raković, Iqbal, Li, Fan, Singh, Surendrannair & Gašević (2023) proved that spoken language does not manifest sentences. An oral utterance contains syntactic clauses but is not a sentence.…”