Computerized Process-Oriented Dynamic Testing of Children’s Ability to Reason by Analogy Using Log Data

Abstract

The study investigated the value of process data obtained from a group-administered computerized dynamic test of analogical reasoning, consisting of a pretest-training-posttest design. We sought to evaluate the effects of training on processes and performance, and the relationships between process measures and performance on the dynamic test. Participants were N = 86 primary school children ( Mage = 8.11 years, SD = 0.63). The test consisted of constructed-response geometrical analogy items, requiring several actions to construct an answer. Process data enabled scoring of the total time, the time taken for initial planning of the task, the time taken for checking the answer that was provided, and variation in solving time. Training led to improved performance compared to repeated practice, but this improvement was not reflected in task-solving processes. Almost all process measures were related to performance, but the effects of training or repeated practice on this relationship differed widely between measures. In conclusion, the findings seemed to indicate that investigating process indicators within computerized dynamic testing of analogical reasoning ability provided information about children’s learning processes, but that not all processes were affected in the same way by training.

Process Data in Computer-Based Assessment

“…In the present issue, the study by Veerbeek and Vogelaar (2023) provides an example of how the item display and the test-taker actions play a fundamental role in the segmentation and interpretation of different time-based measures to track cognitive and metacognitive processes in the context of analogical reasoning tasks. Likewise, Hahnel and colleagues (2023) and Drake and colleagues (2023) employ complex tasks (i.e., web-based search and shopping in a virtual supermarket) that allow for the segmentation of log file-recorded events and a theory-driven definition of behavioral measures.…”

“…For reliable and valid conclusions, a coherent theoretical framework that links the collected data to specific process indicators and the intended measurement of a well-defined theoretical construct is essential (Zumbo et al, 2023). A strong theoretical framework should, thus, allow to identify relevant processes and indicative parameters and provide information on how to collect and analyze the data to test hypotheses regarding construct-relevant processes and support assumptions regarding their interpretation (see, e.g., Drake et al, 2023; Hahnel et al, 2023; Veerbeek & Vogelaar, 2023). The more complex the base of available parameters (e.g., eye-tracking recordings), the more theoretical guidance is required to select appropriate indicators from the diverse range of available data.…”

“…Accordingly, researchers that contributed to the present issue have worked with and reflected on different types of measures and process data indicators. Specifically, behavioral log data from computer-based assessments, including action sequences and pauses (Arslan et al, 2023), time-stamped indicators based on mouse clicks, touch-device input, or keystrokes (Drake et al, 2023;Hahnel et al, 2023;Veerbeek & Vogelaar, 2023), and mouse cursor movements (Pokropek et al, 2023) have been the center of interest in this special issue. Moreover, Koutsogiorgi and Michaelides (2023) analyzed parameters derived from eye movement recordings.…”

“…We believe that we have achieved this ambitious goal by assembling seven excellent research articles in this special issue that illuminate how different types of process data in assessment tasks can be employed in favor of supporting certain diagnostic inferences. Two theoretical groundwork papers (Arslan et al, 2023; Zumbo et al, 2023) and five empirical papers (Drake et al, 2023; Hahnel et al, 2023; Koutsogiorgi & Michaelides, 2023; Pokropek et al, 2023; Veerbeek & Vogelaar, 2023) draw a picture of the crucial role of a strong theoretical foundation and rigorous analysis in making valid inferences from process data, while acknowledging their limitations.…”

“…Abstracting from the specific type of data, the special issue research promotes a variety of targeted constructs. Contributions focused on cognitive and metacognitive (Arslan et al, 2023; Hahnel et al, 2023; Veerbeek & Vogelaar, 2023; Zumbo et al., 2023) as well as self-regulatory processes (Drake et al, 2023) while individuals worked on assessment tasks. Furthermore, two papers investigated careless response behavior (Pokropek et al, 2023) and response tendencies to positive and negative worded items (Koutsogiorgi & Michaelides, 2023) when answering standardized scales.…”