JATOBÁ, Victor Miranda Gonçalves. A personalized approach to the item selection process in Computerized Adaptive Testing. 2019. 59 p. Dissertation
Computerized Adaptive Testing (CAT) based on Item Response Theory allows more accurate assessments with fewer questions than the classic P&P test (Paper and Pencil). Studies showed that CAT using the Fisher Information selection rule reduced the size of the Mathematics and its Technologies ENEM test by 26.6% in relation to the P&P test, which has 45 items. However, the impact of the use of different item selection rules on the estimation of the examinees scores is unknown. The objective of this work is to analise this impact. The results show that the size of this test can be reduzed more with the use of other item selection rules without significant loss of accuracy in the estimation of the proficiency level.Resumo. Testes Adaptativos Computadorizados (CAT), baseados na Teoria de Resposta ao Item, permitem fazer testes mais precisos com um menor número de questões que a prova clássica P&P (Paper and Pencil). Estudos mostraram que CAT, utilizando a regra de seleção de Informação de Fisher, reduziu em 26,6% o tamanho da prova de Matemática e suas Tecnologias do ENEM de 2012 em relação a prova P&P de 45 itens. Entretanto, nãoé conhecido o impacto na estimativa dos escores dos respondentes no uso de diferentes regras de seleção de itens. O objetivo deste trabalhoé analisar esse impacto. Os resultados mostram que o tamanho dessa prova pode ser reduzido ainda mais com o uso de outras regras de seleção de itens, sem perda significativa da precisão na estimativa dos escores dos respondentes.
Abstract. Computerized Adaptive Testing (CAT) based on Item Response Theory (IRT) allows more accurate assessments and shows advantages when incorporated into e-learning environments. However, due the solutions multiplicity, this knowledge is fragmented in the literature, what makes it difficult to understand the challenges faced in the area. In this sense, the goal of this work is to understand and to characterize e-learning systems based on CAT and IRT. To this, a Systematic Review of the literature was performed where four research questions were analyzed in 9 papers selected by the defined protocol criteria. Resumo. Testes Adaptativos Computadorizados (CAT) baseados na Teoria deResposta ao Item (IRT) permitem realizar avaliações mais precisas e mostram vantagens quando incorporados a ambientes de aprendizado virtual (elearning). Entretanto, devidoà multiplicidade de soluções, este conhecimento encontra-se fragmentado na literatura, o que dificulta o entendimento dos dissensos e desafios enfrentados naárea. Neste sentido, este trabalho buscou entender e caracterizar os estudos que usam CAT e IRT que estão inseridos em sistemas e-learning. Para tal, uma revisão sistemática da literatura foi realizada, na qual quatro questões de pesquisa foram analisadas em 9 artigos previamente selecionados pelos critérios estabelecidos no protocolo de revisão.
Computerized adaptive testing (CAT) based on item response theory allows more accurate assessments with fewer questions than the classic paper and pencil (P&P) test. Nonetheless, the CAT construction involves some key questions that, when done properly, can further improve the accuracy and efficiency in estimating the examinees’ abilities. One of the main questions is in regard to choosing the item selection rule (ISR). The classic CAT makes exclusive use of one ISR. However, these rules have differences depending on the examinees’ ability level and on the CAT stage. Thus, the objective of this work is to reduce the dichotomous test size which is inserted in a classic CAT with no significant loss of accuracy in the estimation of the examinee’s ability level. For this purpose, we analyze the ISR performance and then build a personalized item selection process in CAT considering the use of more than one rule. The case study in Mathematics and its Technologies test of the ENEM 2012 shows that the Kullback-Leibler information with a posterior distribution (KLP) has better performance in the examinees’ ability estimation when compared with Fisher information (F), Kullback-Leibler information (KL), maximum likelihood weighted information (MLWI), and maximum posterior weighted information (MPWI) rules. Previous results in the literature show that CAT using KLP was able to reduce this test size by 46.6% from the full size of 45 items with no significant loss of accuracy in estimating the examinees’ ability level. In this work, we observe that the F and the MLWI rules performed better on early CAT stages to estimate examinees’ proficiency level with extreme negative and positive values, respectively. With this information, we were able to reduce the same test by 53.3% using the personalized item selection process, called ALICAT, which includes the best rules working together.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.