Integrating automatic question generation with computerised adaptive test

Abstract

The present study focuses on the integration of an automatic question generation (AQG) system and a computerised adaptive test (CAT). We conducted two experiments. In the first experiment, we administered sets of questions to English learners to gather their responses. We further used their responses in the second experiment, which is a simulation-based experiment of the AQG and CAT integration. We proposed a method to integrate them with a predetermined item difficulty that enables to integrate AQG and CAT without administering the items in a pretesting. The result showed that all CAT simulations performed better than the baseline, a linear test, in estimating the test taker's true proficiency.

Computerized adaptive testing in educational electronic environment of maritime higher education institutions

“…The study of Susanti, Tokunaga and Nishikawa [40] experimentally proved computerised adaptive tests' advantages comparing with the baseline, a linear test.…”

Design of Computerized Adaptive Testing Module into Our Dynamic Adaptive Hypermedia System

“…On the other hand, the authors in [4] developed an adaptive testing system to assess learners' performance using IRT with the Four-Parameter Logistic model (4PL). Also, in [3], the team studied the possibility of integrating an automatic question generation system (AQG) and a computerized adaptive testing (CAT) by predetermining the difficulty of the items without the need to administer them in a pretesting. In 2016, Hoang Tieu Binh [14] proposed to combine IRT and K-Means to assess learners' ability in an e-learning system to rank their level in 10 pre-defined groups.…”

“…Unlike linear tests, CAT works by adapting a test to each candidate, question by question, according to their answers and ability level. If the candidate answers a question correctly, the next question will be more challenging to get as close as possible to his/her level of competence and vice versa [3]. The process of administering CAT is illustrated in figure 1 and has the following steps [1][3] [24]: ─ The test starts with estimating the initial ability level of the candidate.…”

“…CAT is recognized as the next generation of assessments [2]. A CAT is a test delivered by a computer, where the selection of the questions given to the test candidates and the criterion to stop the test dynamically change according to the candidate's competence [3]. Computerized Adaptive Testing (CAT), in contrast to the traditional Paper and Pencil Test (PPT), is a process of test administration in which each student receives a tailored assessment based on his/her skills and abilities [1][3] [4].…”

Artificial Intelligence-Generated and Human Expert-Designed Vocabulary Tests: A Comparative Study

“…Although there are many studies on automatic test item generators for English language testing, the implications for testing Chinese as a second language are yet to be explored. Artificial Intelligence (AI) technologies, which are receiving increased attention in the field of automatic vocabulary item generation, can fill this gap for three reasons (e.g., Susanti et al, 2020; Ulum, 2020): firstly, both selected- and constructed- response formats can be generated with the application of NLP: (1) cloze items (Sakaguchi et al, 2013), (2) multiple-choice vocabulary items (e.g., Aldabe et al, 2006; Hoshino & Nakagawa, 2005), and (3) error correction items (e.g., Aldabe et al, 2006). Secondly, NLP technologies have the potential to create a larger number of distractors for multiple-choice questions (MCQs) in a short period of time using the four approaches as follows: (1) the corpus-based approach (e.g., Aldabe & Maritxalar, 2010), (2) the graph-based approach (e.g., Papasalouros et al, 2008), (3) Word2vec (e.g., Mikolov et al, 2013) and (4) visual similarity (e.g., Jiang & Lee, 2017).…”