Fine Grain Synthetic Educational Data: Challenges and Limitations of Collaborative Learning Analytics

Flanagan, Brendan; Majumdar, Rwitajit; Ogata, Hiroaki

doi:10.1109/access.2022.3156073

Cited by 8 publications

(3 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Techniques like the synthetic minority oversampling technique (SMOTE), adaptive synthetic (ADASYN), and its derivatives aim to counteract these with synthetic data generation [26][27][28][29], but their effectiveness wanes for scant datasets. Although a variety of synthetic data tools and research exist [30][31][32][33][34][35], their applicability is often curtailed when facing complex domain-specific data, particularly in predicting student outcomes [35][36][37]. Navigating this landscape, large language models (LLMs) have made a compelling case as tools for text data augmentation.…”

Section: Augmenting Mathematical Self -Explanations Using Large Langu...mentioning

confidence: 99%

Enhancing Automated Scoring of Math Self-Explanation Quality Using LLM-Generated Datasets: A Semi-Supervised Approach

Nakamoto,

Flanagan,

Yamauchi

et al. 2023

Computers

Self Cite

View full text Add to dashboard Cite

In the realm of mathematics education, self-explanation stands as a crucial learning mechanism, allowing learners to articulate their comprehension of intricate mathematical concepts and strategies. As digital learning platforms grow in prominence, there are mounting opportunities to collect and utilize mathematical self-explanations. However, these opportunities are met with challenges in automated evaluation. Automatic scoring of mathematical self-explanations is crucial for preprocessing tasks, including the categorization of learner responses, identification of common misconceptions, and the creation of tailored feedback and model solutions. Nevertheless, this task is hindered by the dearth of ample sample sets. Our research introduces a semi-supervised technique using the large language model (LLM), specifically its Japanese variant, to enrich datasets for the automated scoring of mathematical self-explanations. We rigorously evaluated the quality of self-explanations across five datasets, ranging from human-evaluated originals to ones devoid of original content. Our results show that combining LLM-based explanations with mathematical material significantly improves the model’s accuracy. Interestingly, there is an optimal limit to how many synthetic self-explanation data can benefit the system. Exceeding this limit does not further improve outcomes. This study thus highlights the need for careful consideration when integrating synthetic data into solutions, especially within the mathematics discipline.

show abstract

Section: Augmenting Mathematical Self -Explanations Using Large Langu...mentioning

confidence: 99%

Enhancing Automated Scoring of Math Self-Explanation Quality Using LLM-Generated Datasets: A Semi-Supervised Approach

Nakamoto,

Flanagan,

Yamauchi

et al. 2023

Computers

Self Cite

View full text Add to dashboard Cite

show abstract

“…Although there's an abundance of generic synthetic data tools and those crafted for spatiotemporal data [31,32], their effectiveness often dwindles when faced with complex, domain-centric data architectures. In learning analytics, the application of synthetic data often stumbles, particularly in the prediction of student outcomes [33][34][35]. LLMs, in light of their recent advancements, are gaining recognition as powerful agents for text data augmentation.…”

Section: Augmenting Mathematical Self -Explanations Using Large Langu...mentioning

confidence: 99%

Enhancing Automated Scoring of Math Self-Explanation Quality using LLM-Generated Datasets: A Semi-Supervised Approach

NAKAMOTO,

Flanagan,

Yamauchi

et al. 2023

Preprint

View full text Add to dashboard Cite

In the realm of mathematics education, self-explanation stands as a crucial learning mechanism, allowing learners to articulate their comprehension of intricate mathematical concepts and strategies. As digital learning platforms grow in prominence, there are mounting opportunities to collect and utilize mathematical self-explanations. However, these opportunities are met with challenges in automated evaluation. Automatic scoring of mathematical self-explanations is crucial for preprocessing tasks, including the categorization of learner responses, identification of common misconceptions, and the creation of tailored feedback and model solutions. Nevertheless, this task is hindered by the dearth of ample sample sets. Our research introduces a semi-supervised technique using the Language Learning Model (LLM), specifically its Japanese variant, to enrich datasets for the automated scoring of mathematical self-explanations. We rigorously evaluated the quality of self-explanations across five datasets, ranging from human-evaluated originals to ones devoid of original content. Our results show that combining LLM-based explanations with mathematical material significantly improves the model's accuracy. Interestingly, there's an optimal limit to how much synthetic self-explanation data can bene-fit the system. Exceeding this limit doesn't further improve outcomes. This study thus highlights the need for careful consideration when integrating synthetic data into solutions, especially within the mathematics discipline.

show abstract

“…The consent propensity differs between student subpopulations by sending our email prompt to a sample of 4,000 students at out institution stratified by gender and ethnicity (Li et al, 2022a). Participants submitted their models as docker containers for evaluation and ranking on holdout synthetic data (Flanagan et al, 2022). The diversified contexts of LA, with the major ones being tertiary education and online learning (Wong et al, 2022).…”

Section: Introductionmentioning

confidence: 99%

The Bayesian CNN-LSTM classification model to predict and evaluate learner’s performance

Sudhamathy,

Valliammal

2023

Int. J. Appl. Sci. Eng.

View full text Add to dashboard Cite

Learning analytics (LA) is a research domain that leverages the analysis of data from the learning process to gain a deeper understanding and enhance learning outcomes. To classify learner performance, a model has been proposed that combines various deep learning techniques, including convolutional neural network (CNN), Long Short-Term Memory (LSTM), and Bayesian models. The integration of these approaches aims to improve the accuracy and effectiveness of performance classification. CNN is used for capturing the local information and LSTM neural network is used for the long-distance dependencies. The effective classification of learners' performance is achieved by combining the strengths of CNN and LSTM, along with the integration of a Bayesian deep learning model. The performance of the proposed model is estimated using the metrics like Accuracy, Precision, Recall and F1-Score. The model showed improvements in Accuracy, Precision, Recall and F1-Score are 98.18%, 97.09%, 96.38% and 95.35% respectively. The proposed model is compared with another existing model such as LSTM and collaborative machine learning (ML) models in terms of performance metrics. The proposed method attained accuracy of 98.18% which is higher than other existing models.

show abstract

Fine Grain Synthetic Educational Data: Challenges and Limitations of Collaborative Learning Analytics

Cited by 8 publications

References 48 publications

Enhancing Automated Scoring of Math Self-Explanation Quality Using LLM-Generated Datasets: A Semi-Supervised Approach

Enhancing Automated Scoring of Math Self-Explanation Quality Using LLM-Generated Datasets: A Semi-Supervised Approach

Enhancing Automated Scoring of Math Self-Explanation Quality using LLM-Generated Datasets: A Semi-Supervised Approach

The Bayesian CNN-LSTM classification model to predict and evaluate learner’s performance

Contact Info

Product

Resources

About