Investigating the Viability of Alternating Model Tree As An Item Selection Algorithm for Constructing Computerized Adaptive Psychological Testing

Abstract

Computerized adaptive testing (CAT) is a computer-administered test where the next question for estimating the examinee's trait level is selected depending on his or her reponses to the previous items, resulting in tailored testing for each individual examinee. A defining feature of CAT stems from its item selection algorithms, among which both research interest and practical applications of decision-tree based CAT (DT-based CAT) have been rising recently. In the field of machine learning, however, it is well known that decision-trees, as a form of predictive models with simple and interpretable tree structures, can be vulnerable to the problem of overfitting or the problem of creating overly complex trees that do not generalize to newly observed data. Among various ensemble techniques developed to adequately address this problem, we the authors paid attention to the Alternating Model Tree (AMT) due to its interpretable tree-like structure. The purpose of this article is to investigate the viability of the Alternating Model Tree (AMT) as an item selection algorithm for constructing CAT. To this end, we first presented a detailed exposition of how AMT-based CAT can be constructed and then compared its performance with DT-based CAT using two sets of publicly available psychological test scores. The results provided supportive evidence that AMT-based CAT is viable, and that AMT-based CAT can predict test scores at least as accurate as DT-based CAT does. Based on our findings, we discuss implications, limitations, and directions of future studies.