Deep Knowledge Tracing with Transformers

Pu, Shi; Yudelson, Michael; Ou, Lu; Huang, Yuchi

doi:10.1007/978-3-030-52240-7_46

Cited by 30 publications

(16 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Selected approach is GKT [22]. • Attention based models: dependence between interactions is captured by the attention mechanism and its variants [10,24,26,47]. Selected approaches are AKT [10], SAKT [23] and SAINT [7].…”

Section: Representative Dlkt Methodsmentioning

confidence: 99%

“…Similar to all existing DLKT research works [1,4,10,11,14,15,20,21,22,24,25,26,29,34,37,44,45,46,47], we use the area under the receiver operating characteristics curve (AUC) as the main metric to evaluate the performance of DLKT models on predicting binary-valued future learner responses to either questions or KCs. Meanwhile, we also include detailed model performance in terms of accuracy in Appendix.…”

Section: Evaluation Metricmentioning

confidence: 99%

See 1 more Smart Citation

pyKT: A Python Library to Benchmark Deep Learning based Knowledge Tracing Models

Liu¹,

Liu²,

Huang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Knowledge tracing (KT) is the task of using students' historical learning interaction data to model their knowledge mastery over time so as to make predictions on their future interaction performance. Recently, remarkable progress has been made of using various deep learning techniques to solve the KT problem. However, the success behind deep learning based knowledge tracing (DLKT) approaches is still left somewhat mysterious and proper measurement and analysis of these DLKT approaches remain a challenge. First, data preprocessing procedures in existing works are often private and/or custom, which limits experimental standardization. Furthermore, existing DLKT studies often differ in terms of the evaluation protocol and are far away real-world educational contexts. To address these problems, we introduce a comprehensive python based benchmark platform, PYKT, to guarantee valid comparisons across DLKT methods via thorough evaluations. The PYKT library consists of a standardized set of integrated data preprocessing procedures on 7 popular datasets across different domains, and 10 frequently compared DLKT model implementations for transparent experiments. Results from our fine-grained and rigorous empirical KT studies yield a set of observations and suggestions for effective DLKT, e.g., wrong evaluation setting may cause label leakage that generally leads to performance inflation; and the improvement of many DLKT approaches is minimal compared to the very first DLKT model proposed by Piech et al. [25]. We have open sourced PYKT and our experimental results at https://pykt.org/. We welcome contributions from other research groups and practitioners.

show abstract

Section: Representative Dlkt Methodsmentioning

confidence: 99%

Section: Evaluation Metricmentioning

confidence: 99%

pyKT: A Python Library to Benchmark Deep Learning based Knowledge Tracing Models

Liu¹,

Liu²,

Huang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…First, we acknowledge that there are a wide variety of knowledge tracing models, including newer ones, that were not included in the empirical evaluation. For example, Pu et al (Pu et al, 2020) and Choi et. al have proposed Transformer-based DLKT models of which the latter has been further extended by Shin et al (Shin et al, 2021), Pandey and Srivastava (Pandey and Srivastava, 2020) have also proposed a Transformer DLKT model (RKT) with the addition of including contextrelated information such as the textual content of the questions as input, Liu et al proposed an exercise enhanced Bi-Directional LSTM with attention model called EKT which leverages both exercise and concept (skill) information, Ghosh et al (Ghosh et al, 2020) developed an attention based DLKT model that fuses DLKT with IRT, Nakagawa et al (Nakagawa et al, 2019) proposed a graph based DLKT model and Song et al (Song et al, 2021) further proposed another graph based DLKT model which, like EKT, models both exercise and skill relations, Yudelson has examined Elo-rating based models for estimating student knowledge (Yudelson, 2019), and Ghosh et al have developed option tracing where the exact choice a student makes is predicted instead of just answer correctness (Ghosh et al, 2021).…”

Section: Limitations Of Work -This Work Is Not Perfect Eithermentioning

confidence: 99%

“…Knowledge tracing (KT) is a student modeling task where knowledge or skill is estimated based on a trace of interactions with learning activities, such as course exercises. While new knowledge tracing models and modifications of existing ones are presented regularly (Yudelson et al, 2013;Piech et al, 2015;Zhang et al, 2017;Yeung and Yeung, 2018;Abdelrahman and Wang, 2019;Nakagawa et al, 2019;Pu et al, 2020), it is not always clear what the actual factors that contribute to the performance of the proposed models are (Wilson et al, 2016;Lipton and Steinhardt, 2018). To what extent do the results depend on the proposed model algorithms and the used data, or are the optimizations the actual key to the results?…”

Section: Introductionmentioning

confidence: 99%

Empirical Evaluation of Deep Learning Models for Knowledge Tracing: Of Hyperparameters and Metrics on Performance and Replicability

Sarsa¹,

Leinonen²,

Hellas³

2021

Preprint

View full text Add to dashboard Cite

Knowledge tracing models, which are used to estimate students' ability or knowledge based on data collected from students' work on learning-related tasks, are a widely studied area within the educational data mining domain. In this work, we review and evaluate a body of deep learning knowledge tracing (DLKT) models with openly available and widely-used data sets, and with a novel data set of students learning to program. The evaluated DLKT models have been re-implemented for assessing reproducibility of previously reported results as well as for assessing the level of detail with which the models and their evaluations have been reported in previously published articles. The DLKT models are tested with different input and output layer variations found in the compared models that are independent of the main architectures of the models and also with different maximum attempt count options. Several metrics to compare and contrast the results are used to reflect on the quality and appropriateness of the evaluated knowledge tracing models. The evaluated knowledge tracing models include Vanilla-DKT, two variants of Long Short-Term Memory Deep Knowledge Tracing (LSTM-DKT), two variants of Dynamic Key-Value Memory Network (DKVMN), and Self-Attentive Knowledge Tracing (SAKT). We evaluate logistic regression, Bayesian Knowledge Tracing (BKT) and simple non-learning models as baselines. Our empirical evaluation suggests that while the DLKT models with tuned hyperparameters in general outperform non deep learning based models, the relative differences between the DLKT models are subtle and often vary between datasets. Specifically, we observe that no model consistently outperforms all other models in all datasets. Our results also show that simple non-learning models such as mean prediction can yield better performance than more sophisticated knowledge tracing models, especially in terms of accuracy, in some datasets. Further, our metric and hyperparameter analysis shows that the metric used to select the best model hyperparameters has a noticeable effect on the performance of the models, and also that some metrics appear more favorable than others for certain models. We also study the effect of input and output layer variations on model performance, and analyze the impact of filtering out long attempt sequences that has been implicitly and explicitly used in some studies. We further discuss the effect of non-model properties such as randomness and hardware on model performance, and finally, we discuss model performance replicability and related issues including pitfalls and suggest practices for future work. Our model implementations, evaluation code, and data are published as a part of this work.

show abstract

“…e dynamic key-value memory network (DKVMN) uses a static memory called key and a dynamic memory called value to discover latent relations between exercises and knowledge concepts [10,11]. Self-attentive knowledge tracing (SAKT) proposes a self-attention-based KT model to model the students' knowledge state, with exercises as attention queries and students' past interactions as attention keys/values [3,[12][13][14][15].…”

Section: Introductionmentioning

confidence: 99%

NTM-Based Skill-Aware Knowledge Tracing for Conjunctive Skills

Qiang

Sun³

et al. 2022

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

Knowledge tracing (KT) is the task of modelling students’ knowledge state based on their historical interactions on intelligent tutoring systems. Existing KT models ignore the relevance among the multiple knowledge concepts of a question and characteristics of online tutoring systems. This paper proposes a neural Turing machine-based skill-aware knowledge tracing (NSKT) for conjunctive skills, which can capture the relevance among the knowledge concepts of a question to model students’ knowledge state more accurately and to discover more latent relevance among knowledge concepts effectively. We analyze the characteristics of the three real-world KT datasets in depth. Experiments on real-world datasets show that NSKT outperforms the state-of-the-art deep KT models on the AUC of prediction. This paper explores details of the prediction process of NSKT in modelling students’ knowledge state, as well as the relevance of knowledge concepts and conditional influences between exercises.

show abstract

Deep Knowledge Tracing with Transformers

Cited by 30 publications

References 4 publications

pyKT: A Python Library to Benchmark Deep Learning based Knowledge Tracing Models

pyKT: A Python Library to Benchmark Deep Learning based Knowledge Tracing Models

Empirical Evaluation of Deep Learning Models for Knowledge Tracing: Of Hyperparameters and Metrics on Performance and Replicability

NTM-Based Skill-Aware Knowledge Tracing for Conjunctive Skills

Contact Info

Product

Resources

About