Feature Enhanced Capsule Networks for Robust Automatic Essay Scoring

Sharma, Arushi; Kabra, Anubha; Kapoor, Rajiv

doi:10.1007/978-3-030-86517-7_23

Cited by 5 publications

(5 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, LLMs-based innovations have achieved high scores of quadratic weighted kappa (QWK) in essay scoring, specifically for off-topic (QWK = 0.80), gibberish (QWK = 0.80), and paraphrased answers (QWK = 0.94), indicating substantial to almost perfect agreements with human raters (Doewes & Pechenizkiy, 2021). Similar performances on essay scoring have been observed in several other studies (eg, 0.80 QWK in (Beseiso et al, 2021) and 0.81 QWK in (Sharma et al, 2021)). Likewise, LLMs-based innovations' performances on automatic short-answer grading were also highly correlated with human ratings (Pearson's correlation between 0.75 to 0.82) (Ahmed et al, 2022;Sawatzki et al, 2022).…”

Section: Performancesupporting

confidence: 88%

Practical and ethical challenges of large language models in education: A systematic scoping review

Yan,

Sha,

Zhao

et al. 2023

Brit J Educational Tech

View full text Add to dashboard Cite

Educational technology innovations leveraging large language models (LLMs) have shown the potential to automate the laborious process of generating and analysing textual content. While various innovations have been developed to automate a range of educational tasks (eg, question generation, feedback provision, and essay grading), there are concerns regarding the practicality and ethicality of these innovations. Such concerns may hinder future research and the adoption of LLMs‐based innovations in authentic educational contexts. To address this, we conducted a systematic scoping review of 118 peer‐reviewed papers published since 2017 to pinpoint the current state of research on using LLMs to automate and support educational tasks. The findings revealed 53 use cases for LLMs in automating education tasks, categorised into nine main categories: profiling/labelling, detection, grading, teaching support, prediction, knowledge representation, feedback, content generation, and recommendation. Additionally, we also identified several practical and ethical challenges, including low technological readiness, lack of replicability and transparency and insufficient privacy and beneficence considerations. The findings were summarised into three recommendations for future studies, including updating existing innovations with state‐of‐the‐art models (eg, GPT‐3/4), embracing the initiative of open‐sourcing models/systems, and adopting a human‐centred approach throughout the developmental process. As the intersection of AI and education is continuously evolving, the findings of this study can serve as an essential reference point for researchers, allowing them to leverage the strengths, learn from the limitations, and uncover potential research opportunities enabled by ChatGPT and other generative AI models. Practitioner notesWhat is currently known about this topic Generating and analysing text‐based content are time‐consuming and laborious tasks. Large language models are capable of efficiently analysing an unprecedented amount of textual content and completing complex natural language processing and generation tasks. Large language models have been increasingly used to develop educational technologies that aim to automate the generation and analysis of textual content, such as automated question generation and essay scoring. What this paper adds A comprehensive list of different educational tasks that could potentially benefit from LLMs‐based innovations through automation. A structured assessment of the practicality and ethicality of existing LLMs‐based innovations from seven important aspects using established frameworks. Three recommendations that could potentially support future studies to develop LLMs‐based innovations that are practical and ethical to implement in authentic educational contexts. Implications for practice and/or policy Updating existing innovations with state‐of‐the‐art models may further reduce the amount of manual effort required for adapting existing models to different educational tasks. The reporting standards of empirical research that aims to develop educational technologies using large language models need to be improved. Adopting a human‐centred approach throughout the developmental process could contribute to resolving the practical and ethical challenges of large language models in education.

show abstract

Section: Performancesupporting

confidence: 88%

Practical and ethical challenges of large language models in education: A systematic scoping review

Yan,

Sha,

Zhao

et al. 2023

Brit J Educational Tech

View full text Add to dashboard Cite

show abstract

“…For example, LLMsbased innovations have achieved high scores of quadratic weighted kappa (QWK) in essay scoring, specifically for off-topic (QWK = 0.80), gibberish (QWK = 0.80), and paraphrased answers (QWK = 0.94), indicating substantial to almost perfect agreements with human raters [15]. Similar performances on essay scoring have been observed in several other studies (e.g., 0.80 QWK in [6] and 0.81 QWK in [56]). Likewise, LLMs-based innovations' performances on automatic short-answer grading were also highly correlated with human ratings (Pearson's correlation between 0.75 to 0.82) [2,46].…”

Section: Practical Challenges -Rq2supporting

confidence: 66%

Practical and Ethical Challenges of Large Language Models in Education: A Systematic Literature Review

Yan¹,

Sha²,

Zhao³

et al. 2023

Preprint

View full text Add to dashboard Cite

Educational technology innovations that have been developed based on large language models (LLMs) have shown the potential to automate the laborious process of generating and analysing textual content. While various innovations have been developed to automate a range of educational tasks (e.g., question generation, feedback provision, and essay grading), there are concerns regarding the practicality and ethicality of these innovations. Such concerns may hinder future research and the adoption of LLMs-based innovations in authentic educational contexts. To address this, we conducted a systematic literature review of 118 peer-reviewed papers published since 2017 to pinpoint the current state of research on using LLMs to automate and support educational tasks. The practical and ethical challenges of LLMs-based innovations were also identified by assessing their technological readiness, model performance, replicability, system transparency, privacy, equality, and beneficence. The findings were summarised into three recommendations for future studies, including updating existing innovations with state-of-the-art models (e.g., GPT-3), embracing the initiative of open-sourcing models/systems, and adopting a human-centred approach throughout the developmental process. These recommendations could support future research to develop practical and ethical innovations for supporting diverse educational tasks and benefiting students, teachers, and institutions.

show abstract

“…The early stage of AES/AWE development is dominated by the traditional machine learning technologies based on Bayes' theorem (Rudner & Liang, 2002), linear regression (Phandi et al, 2015), rank preference learning (Chen & He, 2013;Yannakoudakis et al, 2011), reinforcement learning (Wang et al, 2015), etc. In recent years, the accelerated development of deep learning, especially neural network technology, has also greatly benefited the field of writing evaluation, such as Recurrent Neural Networks (Cai, 2019), Long Short-term Memory (Alikaniotis et al, 2016;Jin et al, 2018;Liu et al, 2019;Taghipour & Ng, 2016), Convolutional Neural Networks (CNN) (Dong & Zhang, 2016;Dong et al, 2017;Farag et al, 2017), BERT-based approach (Sharma et al, 2021). The introduction of attention mechanisms also significantly improved AES SOTA performance (Dong et al, 2017).…”

Section: Literature Reviewmentioning

confidence: 99%

A Multi-Strategy Computer-Assisted EFL Writing Learning System With Deep Learning Incorporated and Its Effects on Learning: A Writing Feedback Perspective

Chen

Bao

Zhang

et al. 2023

Journal of Educational Computing Research

View full text Add to dashboard Cite

Language learning has increasingly benefited from Computer-Assisted Language Learning (CALL) technologies, especially with Artificial Intelligence involved in recent years. CALL in writing learning acknowledged as the core of language learning is being realized by technologies like Automated Writing Evaluation (AWE), and Automated Essay Scoring (AES), which have developed considerably in both computer and language education fields. AWE has effectively enhanced EFL students’ writing performance to some extent, but such technology can only provide an evaluation in the form of scores, the majority of which are based on holistic scoring, resulting in the inability to provide comprehensive and detailed content-based feedback. In order to provide not only the writing multiple trait-specific evaluation scores, but also detailed writing feedback, we proposed a computer-assisted EFL writing learning system incorporating the neural network models and a couple of semantic-based NLP techniques, MsCAEWL, which fully meets the requirements of writing feedback theory, i.e., multiple, continuous, timely, clear, and multi-aspect guidance interactive feedback. The results of comparison experiments with the AWE baseline models and human raters demonstrated the superiority and the high correlation contained by the proposed system. The independent-sample t-test and paired-sample t-test results of the experiments on MsCAEWL effect validation suggested the significant impact of our proposed system in enhancing students’ EFL writing proficiency.

show abstract

Feature Enhanced Capsule Networks for Robust Automatic Essay Scoring

Cited by 5 publications

References 21 publications

Practical and ethical challenges of large language models in education: A systematic scoping review

Practical and ethical challenges of large language models in education: A systematic scoping review

Practical and Ethical Challenges of Large Language Models in Education: A Systematic Literature Review

A Multi-Strategy Computer-Assisted EFL Writing Learning System With Deep Learning Incorporated and Its Effects on Learning: A Writing Feedback Perspective

Contact Info

Product

Resources

About