The relationship between classical item characteristics and item response time on computer-based testing

Abstract

Purpose This study investigated the relationship between the item response time (iRT) and classic item analysis indicators obtained from computer-based test (CBT) results and deduce students’ problem-solving behavior using the relationship. Methods We retrospectively analyzed the results of the Comprehensive Basic Medical Sciences Examination conducted for 5 years by a CBT system in Dankook University College of Medicine. iRT is defined as the time spent to answer the question. The discrimination index and the difficulty level were used to analyze the items using classical test theory (CTT). The relationship of iRT and the CTT were investigated using a correlation analysis. An analysis of variance was performed to identify the difference between iRT and difficulty level. A regression analysis was conducted to examine the effect of the difficulty index and discrimination index on iRT. Results iRT increases with increasing difficulty index, and iRT tends to decrease with increasing discrimination index. The students’ effort is increased when they solve difficult items but reduced when they are confronted with items with a high discrimination. The students’ test effort represented by iRT was properly maintained when the items have a ‘desirable’ difficulty and a ‘good’ discrimination. Conclusion The results of our study show that an adequate degree of item difficulty and discrimination is required to increase students’ motivation. It might be inferred that with the combination of CTT and iRT, we can gain insights about the quality of the examination and test behaviors of the students, which can provide us with more powerful tools to improve them.

Evaluation of usefulness of smart device-based testing: a survey study of Korean medical students

“…With the recent development of smart devices and their high penetration, smart device-based testing (SBT) is gaining prominence as a useful and accurate evaluation method in various areas of education [ 1 , 2 ]. Compared to the paper-pencil test, SBT has many more advantages such as cost reduction and improvement of learning efficiency [ 3 , 4 ], as well as space utilization and mobility, which are cited as shortcomings in CBT. Hence, expectations from SBTs are growing.…”

“…. Compared to the paper-pencil test, SBT has many more advantages such as cost reduction and improvement of learning efficiency [3,4], as well as space utilization and mobility, which are cited as shortcomings in CBT. Hence, expectations from SBTs are growing.…”

Differences in Multiple-Choice Questions of Opposite Stem Orientations Based on a Novel Item Quality Measure

“…However, we included this parameter in the analysis because it correlates with difficulty, 43 associated with difficulty, complexity, and cognitive domain, 44 and may be a surrogate for test-takers' effort or motivation. 45 Considering the claims that NWIs are more difficult because of the extra time and effort needed to correctly read and interpret the question, 10,11 ART is an appropriate, multidimensional measure that should be sensitive to stem orientation. However, ART was not significantly different between control and NWIs, and therefore further supports the difficulty results.…”

Comparing the psychometric properties of two primary school Computational Thinking (CT) assessments for grades 3 and 4: The Beginners' CT test (BCTt) and the competent CT test (cCTt)

“…A point-biserial correlation above 0.15 is recommended, with good items generally having point biserial correlations above 0.25 (Varma, 2006 ). In this article, we consider a threshold of 0.2 , which is commonly employed in the field (Chae et al, 2019 ).…”