The persuasive essays for rating, selecting, and understanding argumentative and discourse elements (PERSUADE) corpus 1.0

Crossley, Scott A.; Baffour, Perpetual; Yu, Tian; Picou, Aigner; Benner, Meg; Boser, Ulrich

doi:10.1016/j.asw.2022.100667

Cited by 12 publications

(10 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The PERSUADE 2.0 corpus originally consists of over 25,000 argumentative essays written by students from 6th to 12th grade in the US for 15 different prompts (Crossley et al 2022). The holistic essay scores, which serve as the ground truth for this study's predictions, were assigned by human raters who underwent training on a scoring rubric employed in the standardized Scholastic Aptitude Test (SAT) in the US.…”

Section: Methods Datasetsmentioning

confidence: 99%

See 1 more Smart Citation

Unveiling the Tapestry of Automated Essay Scoring: A Comprehensive Investigation of Accuracy, Fairness, and Generalizability

Yang,

Raković,

et al. 2024

AAAI

View full text Add to dashboard Cite

Automatic Essay Scoring (AES) is a well-established educational pursuit that employs machine learning to evaluate student-authored essays. While much effort has been made in this area, current research primarily focuses on either (i) boosting the predictive accuracy of an AES model for a specific prompt (i.e., developing prompt-specific models), which often heavily relies on the use of the labeled data from the same target prompt; or (ii) assessing the applicability of AES models developed on non-target prompts to the intended target prompt (i.e., developing the AES models in a cross-prompt setting). Given the inherent bias in machine learning and its potential impact on marginalized groups, it is imperative to investigate whether such bias exists in current AES methods and, if identified, how it intervenes with an AES model's accuracy and generalizability. Thus, our study aimed to uncover the intricate relationship between an AES model's accuracy, fairness, and generalizability, contributing practical insights for developing effective AES models in real-world education. To this end, we meticulously selected nine prominent AES methods and evaluated their performance using seven distinct metrics on an open-sourced dataset, which contains over 25,000 essays and various demographic information about students such as gender, English language learner status, and economic status. Through extensive evaluations, we demonstrated that: (1) prompt-specific models tend to outperform their cross-prompt counterparts in terms of predictive accuracy; (2) prompt-specific models frequently exhibit a greater bias towards students of different economic statuses compared to cross-prompt models; (3) in the pursuit of generalizability, traditional machine learning models (e.g., SVM) coupled with carefully engineered features hold greater potential for achieving both high accuracy and fairness than complex neural network models.

show abstract

Section: Methods Datasetsmentioning

confidence: 99%

“…All these demographic attributes were considered to address RQ2. The details of the dataset are described in (Crossley et al 2023).…”

Section: Methods Datasetsmentioning

confidence: 99%

Unveiling the Tapestry of Automated Essay Scoring: A Comprehensive Investigation of Accuracy, Fairness, and Generalizability

Yang,

Raković,

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…Technology can support feedback generation using automated writing evaluation (AWE; Ngo et al, 2022). Current AWE systems are based on corpora of essays to single writing tasks hand-scored by raters for specific elements related to writing, for example, holistic scores of writing quality (Shermis, 2014) or argumentative elements (Crossley, Baffour, Tian, Picou, Benner & Boser, 2022). These "training data", consisting of texts and human ratings, are then analyzed with machine learning algorithms, which can recognize patterns in the training data and evaluate new texts on the same task based on these patterns (see Ercikan & McCaffrey, 2022, for a description of the strengths and weaknesses of the process).…”

Section: Generating Feedback With Aimentioning

confidence: 99%

Empirische Arbeit: Comparing Generative AI and Expert Feedback to Students’ Writing: Insights from Student Teachers

Jansen,

Höft,

Bahr

et al. 2024

Psychologie in Erziehung Und Unterricht

View full text Add to dashboard Cite

Feedback is crucial for learning complex tasks like writing; yet its creation is time-consuming, often leading to students receiving insufficient feedback. Generative artificial intelligence, particularly Large Language Models (LLMs) like ChatGPT 3.5-Turbo, has been discussed as a solution for providing more feedback. However, there needs to be more evidence that AI-feedback already meets the quality criteria for classroom use, and studies have yet to investigate whether LLM-generated feedback already seems useful to its potential users. In our study, 89 student teachers evaluated the usefulness of feedback for students' argumentative writing, comparing LLM against expert-generated feedback without receiving information about the feedback source. Participants rated LLM-generated feedback as useful for revision in 59 % of texts (compared to 88 % for expert feedback). 23 % of the time, participants preferred to give LLM-generated feedback to students. Our discussion focuses on the conditions in which AI-generated feedback might be effectively and appropriately used in educational settings.

show abstract

“…Although work on medical text has continued, relatively little attention has been devoted to automatic deidentification of educational data. Currently, most educational researchers use human annotators to label PII before releasing educational data sets (Crossley et al, 2022(Crossley et al, , 2023Megyesi et al, 2018). However, the success of deidentification systems in the medical domain suggests that it may be possible to develop automated text deidentification systems for other contexts as well.…”

Section: Approaches To Text Deidentificationmentioning

confidence: 99%

PIILO: an open-source system for personally identifiable information labeling and obfuscation

Holmes,

Crossley,

Sikka

et al. 2023

ILS

View full text Add to dashboard Cite

Purpose This study aims to report on an automatic deidentification system for labeling and obfuscating personally identifiable information (PII) in student-generated text. Design/methodology/approach The authors evaluate the performance of their deidentification system on two data sets of student-generated text. Each data set was human-annotated for PII. The authors evaluate using two approaches: per-token PII classification accuracy and a simulated reidentification attack design. In the reidentification attack, two reviewers attempted to recover student identities from the data after PII was obfuscated by the authors’ system. In both cases, results are reported in terms of recall and precision. Findings The authors’ deidentification system recalled 84% of student name tokens in their first data set (96% of full names). On the second data set, it achieved a recall of 74% for student name tokens (91% of full names) and 75% for all direct identifiers. After the second data set was obfuscated by the authors’ system, two reviewers attempted to recover the identities of students from the obfuscated data. They performed below chance, indicating that the obfuscated data presents a low identity disclosure risk. Research limitations/implications The two data sets used in this study are not representative of all forms of student-generated text, so further work is needed to evaluate performance on more data. Practical implications This paper presents an open-source and automatic deidentification system appropriate for student-generated text with technical explanations and evaluations of performance. Originality/value Previous study on text deidentification has shown success in the medical domain. This paper develops on these approaches and applies them to text in the educational domain.

show abstract

The persuasive essays for rating, selecting, and understanding argumentative and discourse elements (PERSUADE) corpus 1.0

Cited by 12 publications

References 15 publications

Unveiling the Tapestry of Automated Essay Scoring: A Comprehensive Investigation of Accuracy, Fairness, and Generalizability

Unveiling the Tapestry of Automated Essay Scoring: A Comprehensive Investigation of Accuracy, Fairness, and Generalizability

Empirische Arbeit: Comparing Generative AI and Expert Feedback to Students’ Writing: Insights from Student Teachers

PIILO: an open-source system for personally identifiable information labeling and obfuscation

Contact Info

Product

Resources

About