Evaluating the Comparability of Paper‐ and Computer‐Based Science Tests Across Sex and SES Subgroups

Abstract

As access and reliance on technology continue to increase, so does the use of computerized testing for admissions, licensure/certification, and accountability exams. Nonetheless, full computer‐based test (CBT) implementation can be difficult due to limited resources. As a result, some testing programs offer both CBT and paper‐based test (PBT) administration formats. In such situations, evidence that scores obtained from different formats are comparable must be gathered. In this study, we illustrate how contemporary statistical methods can be used to provide evidence regarding the comparability of CBT and PBT scores at the total test score and item levels. Specifically, we looked at the invariance of test structure and item functioning across test administration mode across subgroups of students defined by SES and sex. Multiple replications of both confirmatory factor analysis and Rasch differential item functioning analyses were used to assess invariance at the factorial and item levels. Results revealed a unidimensional construct with moderate statistical support for strong factorial‐level invariance across SES subgroups, and moderate support of invariance across sex. Issues involved in applying these analyses to future evaluations of the comparability of scores from different versions of a test are discussed.

The TIMSS 2019 Item Equivalence Study: examining mode effects for computer-based assessment and implications for measuring trends

“…The above findings meet criteria for evidence that the mathematics and science constructs were unchanged in eTIMSS (APA 1986;DePascale et al 2016;Randall et al 2012;Winter 2010). Therefore, the difference in scores that resulted from the mode effects can be accounted for through appropriate linking procedures and the paperTIMSS and eTIMSS scores can be put on a common scale.…”

“…Addressing the second part of the second research question, the final series of analyses examined differences between paperTIMSS and eTIMSS proficiency scores in relation to student background variables. The results provided additional information about the equivalence of the mathematics and science constructs between modes (Randall et al 2012). If two scores are measuring the same construct, then they should have the same degree of relationship with other related measures (APA 1986;DePascale et al 2016;Winter 2010).…”

Evaluating mode comparability in early elementary grades

“…Research has also been expanded to the item-level analysis as an indication of factor structure by using a combination of analytic techniques such as Classical Test Theory (item difficulties), IRT calibrations and item characteristics curves (ICCs), SIBTEST, and distractor analysis (Bennett et al, 2008;Lin et al, 2016;Poggio et al, 2005;Randall et al, 2012;Welch et al, 2014). According to Keng, McClarty, & Davis (2008), examining mode effects at the item level is helpful because items can provide information about what features need special attention when included in both PPTs and CBTs.…”

“…Csapó, Molnár, and Tóth (2009) Gender. Some studies of gender analysis across modes have reported insignificant results (Bennett et al, 2008;Poggio, Glasnapp, Yang, & Poggio, 2005;Randall et al, 2012). Bennett et al (2008) conducted a comparability study of PPT and CBT versions of the NAEP in mathematics.…”

“…With the widespread use of technology in the classroom, technology has also penetrated the assessment field. Increasingly, many testing programs are using computer-based tests (CBTs), which offer some advantages over paper-and-pencil tests (PPTs), including the use of technology-enhanced items, immediate scoring and reporting, and a richness of interface (Randall, Sireci, Li, & Kaira, 2012;Noyes & Garland, 2008). The introduction of Common Core State Standards and accountability legislation has placed additional attention on the online administration of K-12 assessments (Kim, 2016).…”