From byproduct to design factor: on validating the interpretation of process indicators based on log data

Abstract

International large-scale assessments such as PISA or PIAAC have started to provide public or scientific use files for log data; that is, events, event-related attributes and timestamps of test-takers’ interactions with the assessment system. Log data and the process indicators derived from it can be used for many purposes. However, the intended uses and interpretations of process indicators require validation, which here means a theoretical and/or empirical justification that inferences about (latent) attributes of the test-taker’s work process are valid. This article reviews and synthesizes measurement concepts from various areas, including the standard assessment paradigm, the continuous assessment approach, the evidence-centered design (ECD) framework, and test validation. Based on this synthesis, we address the questions of how to ensure the valid interpretation of process indicators by means of an evidence-centered design of the task situation, and how to empirically challenge the intended interpretation of process indicators by developing and implementing correlational and/or experimental validation strategies. For this purpose, we explicate the process of reasoning from log data to low-level features and process indicators as the outcome of evidence identification. In this process, contextualizing information from log data is essential in order to reduce interpretative ambiguities regarding the derived process indicators. Finally, we show that empirical validation strategies can be adapted from classical approaches investigating the nomothetic span and construct representation. Two worked examples illustrate possible validation strategies for the design phase of measurements and their empirical evaluation.

Process and Product in Computer-Based Assessments

“…This suggests that process and product could readily be integrated into a holistic validation framework. Indeed, this is what Goldhammer and colleagues (2021) propose as they describe a procedure to validate the interpretation and use of process data, supported by appropriate theory, argument, and evidence: “These concepts of validity and validation apply to any indicator-based inferences, regardless of whether product/correctness or process indicators are used. Thus, inferring latent (e.g., cognitive) attributes of the work process from indicators needs to be justifiable.” (Goldhammer et al, 2021, p. 13) Their argument suggests a requirement to validate the process and product data in the same way under a holistic validity framework that includes a plurality of constructs.…”

“…Following the positionality of “construct theory” embraced by Goldhammer and colleagues (2021), this need for clarity of the concept of validity is paramount within a holistic framework where one can conceive of the test-oriented construct(s) and process-oriented constructs. It should be noted that many authors refer to construct validity as the most important characteristic of a test, but it is seldom defined.…”

“…However, its different areas of foci concern different phenomena – some of which may be inferred (e.g., thought processes) and others that can more readily be observed and captured (e.g., behaviors). Similarly, they include second-order explanations’ strategies, approaches’ which might be inferred from a theory of test-taker behavior – what Goldhammer and colleagues (2021) describe as high-level interpretations.…”

“…On the contrary, their potential to inform and improve digital assessments is clear (Goldhammer et al, 2020; Jiao et al, 2021). However, in the absence of systematic and routinized validation, there is a requirement for more explicit arguments and evidence than we might consider necessary for product-based scores (Goldhammer et al, 2021).…”

The uses of process data in large-scale educational assessments

“…There are multiple measures of test engagement. These include questionnaires and surveys such as the Programme for International Student Assessment (PISA) 'effort thermometer' and 'perseverance index' (Eklöf and Knekta, 2017 [32]; Eklöf and Hopfenbeck, 2019 [33]), the use of eye tracking studies (Oranje et al, 2017 [23]; Maddox, 2018 [28]), and rapidly expanding use of log data on item response times and keystrokes (Goldhammer, Martens and Lüdtke, 2017 [65]; Wise, 2017 [37]; Wise, 2019 [66]; Wise, Kuhfeld and Soland, 2019 [58]; Lee and Jia, 2014 [67]; Gneezy et al, 2019 [62]; Kroehne, Deribo and Goldhammer, 2020 [68]; Wise, 2020 [69]).…”