Towards an Appropriate Query, Key, and Value Computation for Knowledge Tracing

Abstract

Knowledge tracing, the act of modeling a student's knowledge through learning activities, is an extensively studied problem in the field of computer-aided education. Armed with attention mechanisms focusing on relevant information for target prediction, recurrent neural networks and Transformer-based knowledge tracing models have outperformed traditional approaches such as Bayesian knowledge tracing and collaborative filtering. However, the attention mechanisms of current state-of-the-art knowledge tracing models share two limitations. Firstly, the models fail to leverage deep self-attentive computations for knowledge tracing. As a result, they fail to capture complex relations among exercises and responses over time. Secondly, appropriate features for constructing queries, keys and values for the self-attention layer for knowledge tracing have not been extensively explored. The usual practice of using exercises and interactions (exercise-response pairs), as queries and keys/values, respectively, lacks empirical support.In this paper, we propose a novel Transformer-based model for knowledge tracing, SAINT: Separated Self-AttentIve Neural Knowledge Tracing. SAINT has an encoder-decoder structure where the exercise and response embedding sequences separately enter, respectively, the encoder and the decoder. The encoder applies self-attention layers to the sequence of exercise embeddings, and the decoder alternately applies selfattention layers and encoder-decoder attention layers to the sequence of response embeddings. This separation of input allows us to stack attention layers multiple times, resulting in an improvement in area under receiver operating characteristic curve (AUC). To the best of our knowledge, this is the first work to suggest an encoder-decoder model for knowledge tracing that applies deep self-attentive layers to exercises and responses separately.We empirically evaluate SAINT on a large-scale knowledge tracing dataset, EdNet, collected by an active mobile education

SAINT+: Integrating Temporal Features for EdNet Correctness Prediction

“…In each self-attention layer of SAKT, each query is an exercise embedding vector, and key and value are interaction embedding vectors. SAINT [2] is the first Transformer based knowledge tracing model which leverages encoder-decoder architecture composed of stacked self-attention layers. Unlike SAKT, SAINT gets separated streams of exercises and responses as inputs where a sequence of exercises are fed to the encoder, and a sequence of encoder outputs and responses are fed to the decoder.…”

“…In this subsection, we give a brief review of SAINT, a Separated Self-AttentIve Neural Knowledge Tracing. We refer the paper [2] for those who want to lean detailed aspects of SAINT. SAINT is a knowledge tracing model based on Transformer [25] The most fundamental part of SAINT is a multi-head attention layer.…”

“…In this paper, we propose SAINT+, a successor of SAINT [2] which enhances knowledge tracing with temporal feature embeddings, and empirically verify the effectiveness of the model on EdNet dataset. SAINT is a Transformer [25] based knowledge tracing model that separately processes information of exercise and student response.…”

Prediction of learning outcomes via clickstream data using machine learning

“…However, unlike the single matrix in MANN, DKVMN uses two matrices a key matrix for latent concepts and value matrix for student mastery level. More recently, transformers have also been applied to train DKTs [225][226][227]. Since they employ a self-attention mechanism, information encoded by transformers may also be interpreted.…”

“…Random forests [174,197,199], k-nearest neighbours [204], Neural networks [177,200,[208][209][210]219], Bayesian networks [197,210,214], Regression models [174,177,178,188,198,205,206,213,216,217], Nave Bayes [174,214], Rule-based systems [197,207,209,221], Decision trees [174, 183, 196-198, 210, 214], Correlational analysis [220], Support vector machines [198,214], Matrix factorization & collaborative filtering [200-203, 211, 212], Cox proportional hazard model [215] Deep knowledge tracing -Deep knowledge tracing [31,222,223], Memory-augmented neural networks [224], Transformers [225][226][227] Chapter 6…”