Invariance of the Response Processes Between Gender and Modes in an Assessment of Reading

Abstract

In this paper, we developed a method to extract item-level response times from log data that are available in computer-based assessments (CBA) and paper-based assessments (PBA) with digital pens. Based on response times that were extracted using only time differences between responses, we used the bivariate generalized linear IRT model framework (B-GLIRT, [1]) to investigate response times as indicators for response processes. A parameterization that includes an interaction between the latent speed factor and the latent ability factor in the cross-relation function was found to fit the data best in CBA and PBA. Data were collected with a within-subject design in a national add-on study to PISA 2012 administering two clusters of PISA 2009 reading units. After investigating the invariance of the measurement models for ability and speed between boys and girls, we found the expected gender effect in reading ability to coincide with a gender effect in speed in CBA. Taking this result as indication for the validity of the time measures extracted from time differences between responses, we analyzed the PBA data and found the same gender effects for ability and speed. Analyzing PBA and CBA data together we identified the ability mode effect as the latent difference between reading measured in CBA and PBA. Similar to the gender effect the mode effect in ability was observed together with a difference in the latent speed between modes. However, while the relationship between speed and ability is identical for boys and girls we found hints for mode differences in the estimated parameters of the cross-relation function used in the B-GLIRT model.Technology-based assessments offer the possibility to collect additional log data, including response times as the amount of time test-takers spend responding to particular questions or tasks. Primarily, response times provide information about test-takers' speed, which-if not considered-might confound comparisons of test-takers' ability [2] due to inter-individual differences in the speed-ability compromise [3]. Response times can allow inferences about response processes [4,5]. Although at the individual level response times are influenced by multiple factors [6], the comparison of characteristics at the group level, such as the between-person relationship of speed

Construct Equivalence of PISA Reading Comprehension Measured With Paper‐Based and Computer‐Based Assessments

“…However, as more than just the (overall) number of missing values seems relevant, further research is necessary to fully understand the empirical and theoretical role of missing value coding in the context of comparisons across modes and over time in terms of trends. As we did not compare test speed between modes, we see the need to investigate time‐related differences between modes in the future (see, e.g., Kroehne, Hahnel & Goldhammer, ).…”

Process Data Analysis in ILSAs

“…Using response times from ILSAs in a joint modeling framework with response accuracy, Kroehne et al (2019) evaluated the invariance of response processes regarding the assessment mode (i.e., computer-based vs. paper-based) and the respondent's gender. With an online publication appearing in 2019, the work of Ulitzsch et al (2020a), in turn, aimed to develop a framework that incorporates respondent's nonresponse behavior to gain a deeper understanding of the processes underlying item omissions in large-scale assessments.…”

Reanalysis of the German PISA Data: A Comparison of Different Approaches for Trend Estimation With a Particular Emphasis on Mode Effects

“…As the use and importance of computers increases in society and education, it seems more relevant to assess competencies in CBA than in PBA mode, and it could therefore be argued that it is not necessarily of interest how a trend would continue if it were based on a PBA. Accordingly, future research should investigate potential sources for mode effects, such as mode-related speed differences (Kroehne et al, 2019b).…”