Evaluation of Automatic Item Generation Utilities in Formative Assessment Application for Korean High School Students

Abstract

The recent interests in research in the assessment field have been rapidly shifting from decision-maker-centered assessments to learner-centered assessments (i.e., diagnostic and/or formative assessments). In particular, it is a very important research topic in this field to analyze how these learner-centered assessments are developed more practical and valid by combining information (or intelligent) and communication technologies (ICT) and psychometric advances, and how these systems contribute to learning. Automatic Item Generation (AIG), which is in the spotlight recently, is a representative example of building a next-generation assessment theory and practice by integrating ICT into psychometrics.AIG has very broad and promising features in mass production, intelligent item calibration and management, especially learner-centered assessment, and so on. However, these claims have not yet been fully validated in real education settings, and it is in dire need of evaluating the utilities of AIG applications in practical educational settings. Based on these needs, the purposes of this research are, firstly, to introduce the benefits and challenges of AIG to properly assess AIG’s utilities. And, secondly, this study provides empirical evaluation results of AIG’s utilities within a formative assessment system for Korean high school students and teachers.The results of this study have important implications in that it is the first empirical study to evaluate the usefulness of AIG in the formation evaluation application of the actual educational environment, Korean high school setting. The results of the theoretical and empirical evaluation of the AIG will also be useful to researchers and practitioners who wish to evaluate the benefits and issues of future AIG-based educational services or theory development.

Feasibility assurance: a review of automatic item generation in medical assessment

“…Bearing this in mind, and since we continually make judgements and decisions regarding medical students based on assessment, there is a need to understand AIG’s strengths and limitations as an assessment tool (Cook et al, 2015 ). This need is related to the fact that AIG applications in educational settings have not yet been validated, as well as the psychometric characteristics of the items generated remain unreported (Choi et al, 2018 ; Gierl et al, 2016 ).…”

Role of Formative Assessment in Improving Students’ Motivation, Engagement, and Achievement: A Systematic Review of Literature

“…Two qualitative studies (Van der Nest, Long, and Engelbrecht 2018; Beesley et al 2018) focused on teachers' professional development while three studies (Brink and Bartz 2017;Cisterna and Gotwals 2018;Lyon, Oláh, and Wylie 2019) focused on the implementation of formative assessment. Other non-experimental but quantitative studies also focused on the implementation and use of formative assessment (Bulunuz et al 2015;Saito and Inoi 2017;Choi, Kim, and Pak 2018;Xiao and Yang 2019). Although this study aimed to find out how specific strategies of formative assessment have been applied in research, twenty-one studies did not focus on a single strategy but focused on formative assessment as a whole.…”

Computerized Item Modeling Practices using Computer Adaptive Formative Assessment Automatic Item Generation System: A Tutorial

“…In particular, there is evidence to suggest that AIG can offer a variety of solutions to help improve assessment practices and address policy related concerns (e.g., exam leakages, teaching to test, financial burden associate with assessments, etc. ; Choi, Kang, Kim, Dardick, & Zhang, 2015;Choi, Kim, & Pak, 2018;Dardick & Choi, 2016).…”

“…The items generated using CAFA AIG have proven to be useful in measuring different types of ability, including knowledge, comprehension, application, and analysis, at many levels. Incorporating such technology-enhanced features in assessments can improve the learning motivation of students and the efficiency of teaching (Choi et al, 2018). In addition, thanks to the mass production capability of AIG, these examples can be clear evidence of the claim that AIG is a sustainable and efficient tool for the transition from paper pencils to genuine digital assessments (Choi, 2017).…”