Process and Product in Computer-Based Assessments

Abstract

There is no consensus among assessment researchers about many of the central problems of response process data, including what is it and what is it comprised of. The Standards for Educational and Psychological Testing ( American Educational Research Association et al., 2014 ) locate process data within their five sources of validity evidence. However, we rarely see a conceptualization of response processes; rather, the focus is on the techniques and methods of assembling response process indices or statistical models. The method often overrides clear definitions, and, as a field, we may therefore conflate method and methodology – much like we have conflated validity and validation ( Zumbo, 2007 ). In this paper, we aim to clear the conceptual ground to explore the scope of a holistic framework for the validation of process and product. We review prominent conceptualizations of response processes and their sources and explore some fundamental questions: Should we make a theoretical and practical distinction between response processes and response data? To what extent do the uses of process data reflect the principles of deliberate, educational, and psychological measurement? To answer these questions, we consider the case of item response times and the potential for variation associated with disability and neurodiversity.

Process Data in Computer-Based Assessment

“…General definitions of process data encompass a range of data types, including, for example, verbal protocols, repeated survey questions, observational data, or physiological measures (see, e.g., Maddox, 2017, 2023; Oranje et al, 2017). Since the mid-twentieth century, process data in this broader sense have gained attention following the early work of researchers such as Lindquist (1951), Ryans and Frederiksen (1951), and Cronbach and Meehl (1955) in the course of validity discussions (see, e.g., Ercikan et al, 2020; Zumbo et al, 2023). Process data have since been valued for their potential to support the validation process of assessment tasks (see, e.g., Cronbach, 1971; Messick, 1989a, 1989b, 1995) in conjunction with theoretical conceptualizations of construct measures and, for example, cognition-oriented item response theory (IRT) and cognitive diagnostic modeling approaches (e.g., Embretson, 1994, 1998; Embretson & Gorin, 2001; Ercikan & Pellegrino, 2017; Gorin, 2006; Leighton & Gierl, 2007a).…”

“…However, it is a long way from collecting a stream of process data and interpreting the data in a construct-relevant manner, as Zumbo and colleagues (2023) discuss in greater detail in their special issue article. Sometimes, process data may not hold up to the expectation of supporting the assessment of a target construct, so a naïve view that tracking a process will inevitably provide useful information for the interpretation of the resulting product scores does not necessarily apply.…”

“…Thus, we may appreciate the potentials of process data but also need to acknowledge their constraints and the specific situations where justifiable inferences can (or cannot) be drawn based on sound validity arguments for their interpretation (see, e.g., Goldhammer et al, 2021). In this regard, Zumbo and colleagues (2023) further highlight that it is important that researchers do not conflict the term process data (i.e., the manifest collection of behavioral data as traces of processes; see also Ercikan et al, 2020) with response processes (i.e., the latent construct and underlying processes we aim to infer; see also Hubley & Zumbo, 2017).…”

“…We believe that we have achieved this ambitious goal by assembling seven excellent research articles in this special issue that illuminate how different types of process data in assessment tasks can be employed in favor of supporting certain diagnostic inferences. Two theoretical groundwork papers (Arslan et al, 2023; Zumbo et al, 2023) and five empirical papers (Drake et al, 2023; Hahnel et al, 2023; Koutsogiorgi & Michaelides, 2023; Pokropek et al, 2023; Veerbeek & Vogelaar, 2023) draw a picture of the crucial role of a strong theoretical foundation and rigorous analysis in making valid inferences from process data, while acknowledging their limitations.…”

“…Abstracting from the specific type of data, the special issue research promotes a variety of targeted constructs. Contributions focused on cognitive and metacognitive (Arslan et al, 2023; Hahnel et al, 2023; Veerbeek & Vogelaar, 2023; Zumbo et al., 2023) as well as self-regulatory processes (Drake et al, 2023) while individuals worked on assessment tasks. Furthermore, two papers investigated careless response behavior (Pokropek et al, 2023) and response tendencies to positive and negative worded items (Koutsogiorgi & Michaelides, 2023) when answering standardized scales.…”