Introducing a Framework to Assess Newly Created Questions with Natural Language Processing

Abstract

Statistical models such as those derived from Item Response Theory (IRT) enable the assessment of students on a specific subject, which can be useful for several purposes (e.g., learning path customization, drop-out prediction). However, the questions have to be assessed as well and, although it is possible to estimate with IRT the characteristics of questions that have already been answered by several students, this technique cannot be used on newly generated questions. In this paper, we propose a framework to train and evaluate models for estimating the difficulty and discrimination of newly created Multiple Choice Questions by extracting meaningful features from the text of the question and of the possible choices. We implement one model using this framework and test it on a real-world dataset provided by CloudAcademy, showing that it outperforms previously proposed models, reducing by 6.7% the RMSE for difficulty estimation and by 10.8% the RMSE for discrimination estimation. We also present the results of an ablation study performed to support our features choice and to show the effects of different characteristics of the questions' text on difficulty and discrimination.

Text-based Question Difficulty Prediction: A Systematic Review of Automatic Approaches

“…4 Most common prediction models (as a percentage of publications studied) interesting is that all the studies that selected RF as a prediction model conducted an algorithm selection experiment to choose the algorithm that performs better on the question difficulty prediction task. It was consistently found across these studies that the RF regressor outperformed other ML algorithms such as Linear regression, SVM, Gaussian processes, fully connected neural networks and Decision Trees (Benedetto et al, 2020a;Xu et al, 2022;Yaneva et al, 2020).…”

“…Paper Citation TF-IDF (Benedetto et al, 2020a(Benedetto et al, , 2020b) (Lin et al, 2015) Readability measures (Benedetto et al, 2020a;Choi & Moon, 2020) (Susanti et al, 2017;Yaneva et al, 2020) (Yaneva et al, 2019) Corpus analysis software (Choi & Moon, 2020;Pandarova et al, 2019) (El Masri et al, 2017Lee et al, 2019) (Beinborn et al, 2014(Beinborn et al, , 2015) (Loukina et al, 2016;Sano, 2015) Word embedding (Benedetto et al, 2021;Xu et al, 2022) (Bi et al, 2021;Loginova et al, 2021) (Susanti et al, 2020;Xue et al, 2020) (Yaneva et al, 2020;Zhou & Tao, 2020) (Yaneva et al, 2019;Yeung et al, 2019) (Cheng et al, 2019;Hsu et al, 2018) (Huang et al, 2017) Ontology-based metrics (Kurdi et al, 2021;Vinu & Kumar, 2020) (Faizan & Lohmann, 2018;Seyler et al, 2017) (Vinu et al, 2016;Vinu & Kumar, 2017) (Alsubait et al, 2016;Vinu & Kumar, 2015) LSTM/ BiLSTM (Lin et al, 2019;Qiu et al, 2019) (Cheng et al, 2019;Gao et al, 2018) Syntax-level Feature Extraction When investigating sources of difficulty in textual questions, textual co...…”

“…The basic intuition here is that more textually complex questions require students to have more advanced language proficiency skills in order to read, understand and answer questions correctly. Determining the linguistic complexity of the question's string is an intuitive difficulty measure that was utilised in a number of studies and is considered one of the basic tasks in NLP (Benedetto et al, 2020a(Benedetto et al, , 2020bSusanti et al, 2017;Yaneva et al, 2019). For this purpose, readability measures were commonly used to produce descriptive statistics that quantify how difficult a text (in our case, a question) is to read.…”

“…Thus, the underlying intuition here is that a question's readability level correlates with its level of difficulty. Benedetto et al (2020a) used these readability indexes (in combination with other features) to measure text complexity of questions by counting features such as the number of words and average word length. However, on their own, the readability measures did not perform as well as other linguistic features such as TF-IDF.…”

Field-testing multiple-choice questions with AI examinees

“…Some methods rely on expert judgement (Beinborn, Zesch, & Gurevych, 2014;Choi & Moon, 2020;Loukina et al, 2016;Settles, LaFlair & Hagiwara, 2020), but these subjective approaches can suffer from poor inter-rater-reliability (i.e., consistency between multiple judges) and replicate-ability (AlKhuzaey et al, 2023;Conejo et al, 2020). Others relied on machine-driven natural language processing (NLP) techniques to predict item difficulty and/or discrimination (Benedetto et al, 2020a(Benedetto et al, , 2020bBenedetto et al, 2021;Yaneva et al, 2019;Zhou & Tao, 2020). However, their level of prediction accuracy is limited, and a simple estimation of item difficulty and discrimination does not capture the comprehensive nature of traditional field-testing.…”