SAINT+: Integrating Temporal Features for EdNet Correctness Prediction

Abstract

We propose SAINT+, a successor of SAINT which is a Transformer based knowledge tracing model that separately processes exercise information and student response information. Following the architecture of SAINT, SAINT+ has an encoder-decoder structure where the encoder applies self-attention layers to a stream of exercise embeddings, and the decoder alternately applies self-attention layers and encoder-decoder attention layers to streams of response embeddings and encoder output. Moreover, SAINT+ incorporates two temporal feature embeddings into the response embeddings: elapsed time, the time taken for a student to answer, and lag time, the time interval between adjacent learning activities. We empirically evaluate the effectiveness of SAINT+ on EdNet, the largest publicly available benchmark dataset in the education domain. Experimental results show that SAINT+ achieves state-of-the-art performance in knowledge tracing with an improvement of 1.25% in area under receiver operating characteristic curve compared to SAINT, the current state-of-the-art model in EdNet dataset. CCS CONCEPTS• Computing methodologies → Artificial intelligence; Neural networks; • Applied computing → Interactive learning environments; • Social and professional topics → Student assessment.

Empirical Evaluation of Deep Learning Models for Knowledge Tracing: Of Hyperparameters and Metrics on Performance and Replicability

“…Multiple more recent approaches appear promising (Nakagawa et al, 2019;Ghosh et al, 2020;Cheng et al, 2020;Pandey and Srivastava, 2020;Oya and Morishima, 2021;Shin et al, 2021;Song et al, 2021), and many claim significant performance improvements, but their results still require verification via replication studies. Also, many of these newer models include slightly different inputs, such as skill and previous correctness as separate inputs , additional time related inputs (Shin et al, 2021) or leveraging both exercise and skill labels (Song et al, 2021), giving rise to the question of whether and how much the older architectures would also benefit from such input additions. We agree with this direction of exploring new inputs since DLKT models are powerful models that are likely to benefit from such additional information.…”

“…These include e.g. DKVMN (Zhang et al, 2017), SKVMN (Abdelrahman and Wang, 2019), DQN (Lee and Yeung, 2019), GNNKT (Nakagawa et al, 2019), SAKT (Pandey and Karypis, 2019), SAINT , SAINT+ (Shin et al, 2021), and JKT (Song et al, 2021), many of which include additional inputs and adjust the layer structure by introducing techniques from the broader machine learning domain.…”

“…For example, Pu et al (Pu et al, 2020) and Choi et. al have proposed Transformer-based DLKT models of which the latter has been further extended by Shin et al (Shin et al, 2021), Pandey and Srivastava (Pandey and Srivastava, 2020) have also proposed a Transformer DLKT model (RKT) with the addition of including contextrelated information such as the textual content of the questions as input, Liu et al proposed an exercise enhanced Bi-Directional LSTM with attention model called EKT which leverages both exercise and concept (skill) information, Ghosh et al (Ghosh et al, 2020) developed an attention based DLKT model that fuses DLKT with IRT, Nakagawa et al (Nakagawa et al, 2019) proposed a graph based DLKT model and Song et al (Song et al, 2021) further proposed another graph based DLKT model which, like EKT, models both exercise and skill relations, Yudelson has examined Elo-rating based models for estimating student knowledge (Yudelson, 2019), and Ghosh et al have developed option tracing where the exact choice a student makes is predicted instead of just answer correctness (Ghosh et al, 2021). Limiting the number of evaluated models was a deliberate choice, as we meticulously re-implemented the algorithms as well as compared and contrasted the available implementations with the details outlined in the respective articles.…”

Bi-CLKT: Bi-Graph Contrastive Learning based Knowledge Tracing

“…• SAINT+ [37] , the first Transformer-based Knowledge Tracing model, is unique in that it introduces exercise information as well as student response information separately, while at the same time it embeds two temporal features, elapsed time and lag time, into the embedding of student response information.…”

Knowledge structure enhanced graph representation learning model for attentive knowledge tracing

“…The elapsed time et strongly evidence a student's proficiency in knowledge and skills. 51 This time is converted to seconds and capped at 500 s. A d′-dimensional latent embedding vector for et k is computed as…”