Computer-Based Testing: Score Equivalence and Testing Administration Mode Preference in a Comparative Evaluation Study

Abstract

The empirical evidences show that two identical Computer-Based Testing (henceforth CBT) and Paper-and-Pencil-Based Testing (henceforth PBT) do not always result in the same scores. Such conclusions are referred to as “the effect of testing administration mode” or “testing mode effect”. Moderators such as individual differences (e.g., prior computer experience or computer attitude) have been investigated [4] to see if they influence test takers’ performance. The Guidelines for Computer-Based Tests and Interpretations [1] recommended eliminating the possible effects of some moderator variables on test takers performance. This research was conducted to provide the required empirical evidences on the existence of distinctive effects caused by changing administration mode from conventional PBT to modern CBT. The relationship between testing mode preference on test takers’ CBT performance was also examined. Two equivalent tests and two questionnaires were used. Using descriptive statistics and ANOVA, the findings demonstrated that two CBT and PBT sets of scores were comparable. Additionally, prior testing mode preference and gender had no significant effect on test takers’ CBT score, and they were not considered the variables that might affect the performance on CBT.

Computerized testing in reading comprehension skill: investigating score interchangeability, item review, age and gender stereotypes, ICT literacy and computer attitudes

“…The CFLT2 was administered to examine the effect of item review on their performance. Although some researchers have concluded that CFLT version of test resulted in lower scores than in paper-based tests (Chen et al, 2011;Clinton, 2019;Delgao et al, 2018;Golan et al, 2018;Lenhard et al, 2017;Rasmusson, 2015), in this research, the results of the participants' testing performance in both PPBT and CFLT1 revealed that there was not any significant difference between the two sets of scores obtained from the two versions of the reading comprehension test that are in line with some studies (Farinosi et al, 2016;Hashemi Toroujeni et al, In Press;Hermena et al, 2017;Khoshsima et al, 2017h;Kong et al, 2018;Meyer et al, 2016;Prisacari & Danielson, 2017). In fact, participants' performances in two conventional paper-based and computerized testing sessions were not different.…”

“…Use of computer as an integral (Stole et al, 2020) and ubiquitous part of every society (Mullis, Martin, Foy, and Hooper, 2017) raised testing productivity by introducing efficacious tools that have some advantages (Shraim, 2019) such as evaluating responses and giving proper prompt feedback in an intelligent environment (Kilickaya, 2013). Popularity of using computer to improve testing approaches and solve largescale assessment problems that is increasingly growing in recent years (Sawada, 2019) caused entirely practical reconstructions in assessment procedures (Khoshsima and Hashemi Toroujeni, 2017h). In particular, Covid-19 identification late 2019 and health crisis caused by Coronavirus outbreak (WHO, 2020) all over the world (Karim & Hasan, 2020) leaded education systems of especially Asian developing countries to face their greatest challenge because their technology infrastructure was not sufficient (Retnawati, 2015) for digitalized education, and investments on technology (ADB, 2018;Sawada, 2019) is still developing in these countries for a widespread use of computer-based teaching, learning and testing as the new approaches (Karim & Hasan, 2020).…”

“…A CFLT version that is highly endorsed by test takers may not be valid, reliable and equivalent to its PPBT counterpart. PPBT and CFLT equivalency means that validity (the degree to which a specific test measures exactly what it claims to measure) (Stobart, 2012) and reliability (the degree to which a test measures consistently and stably what it claims to measure) (Scheerens, Glas, and Thomas, 2005, p. 93) of a test are not violated as a result of transition (Khoshsima and Hashemi Toroujeni, 2017h). Dogan et al, (2020) believe that because transition of PPBT to CFLT is growing due to the universal increasing use of personal computers, there is no controversy against two versions equivalency in most of assessments.…”

“…Some studies report score interchangeability and no statistically significant difference between paper-based and computerized tests (Hashemi Toroujeni et al, in press;Khoshsima and Hashemi Toroujeni, 2017h;Prisacari & Danielson, 2017;Register-Mihalik et al, 2012). Although Ebrahimi, Hashemi Toroujeni and Shahbazi (2019), Hermena et al (2017), Khoshsima et al (2019), Khoshsima and Hashemi Toroujeni (2017h), Porion et al (2016) indicate that two identical computer-based and paperbased tests may result in the same scores; some others reveal different test results (Emerson & MacKay, 2011;Galindo-Aldana et al, 2018;Jerrim, 2016;Jerrim et al, 2018;Kim & Kim, 2013;Washburn, Herman, Stewart, 2017) especially in reading comprehension skill (Clinton, 2019;Delgado et al, 2018;Stole et al, 2020) due to the "Testing Mode Effect." Such empirical findings help testing practitioners decide whether to replace computer-based testing with its identical paper-based test.…”

Computer-based Adaptive Test Development Using Fuzzy Item Response Theory to Estimate Student Ability

“…The use of computer technology to improve the quality of test results has been widely carried out. In the era of advanced technology and information, it is very feasible to conduct computer-based tests [4]. Reference [5][6][7] show that testing using a computer is not only able to produce tests that are fast and accurate but can also increase the effectiveness and efficiency of test implementation and maintenance.…”