Process Data in NAEP: Past, Present, and Future

Abstract

This article reviews how National Assessment of Educational Progress (NAEP) has come to collect and analyze data about cognitive and behavioral processes (process data) in the transition to digital assessment technologies over the past two decades. An ordered five-level structure is proposed for describing the uses of process data. The levels in this hierarchy range from ignoring the processes (i.e., counting only the outcomes), to incorporating process data as auxiliary or essential in addition to the outcome, to modeling the process as the outcome itself, either holistically in a rubric score or in a measurement model that accounts for sequential dependencies. Historical examples of these different uses are described as well as recent results using nontraditional analytical approaches. In the final section, speculative future directions incorporating state-of-the-art technologies and analysis methods are described with an eye toward hard-to-measure constructs such as higher order problem-solving and collaboration.

Assessing Preknowledge Cheating via Innovative Measures: A Multiple-Group Analysis of Jointly Modeling Item Responses, Response Times, and Visual Fixation Counts

“…As De Boeck and Jeon (2019) state, “abilities refer to levels of performance, whereas processes are the activities involved in reaching a performance outcome” (p. 1). Bergner and von Davier (2019) describe the process in process data and its relation to outcome measures in the following ways:…”

“…Although its importance in understanding the complexities of the learning process, notwithstanding integration and modeling of biometric data in practice has been slow because these data must be captured concurrently in real-time with other more conventional data types (Man, 2020). One type of biometric data that is emerging is eye-tracking data (Bergner & von Davier, 2019). Eye-tracking data has been used in various disciplines for some time and different attributes of eye-movement such as visual fixation counts (VFCs) have proven to be conducive in understanding many cognitive processes (Poole et al, 2004).…”

Cognitive Diagnosis Modelling Incorporating Response Times and Fixation Counts: Providing Comprehensive Feedback and Accurate Diagnosis

“…In CDMs, these are primarily used to obtain current knowledge structure. Process data (e.g., RTs and operating steps) refer to the additional information collected about student interactions with the assessment environment, which can be used to reconstruct the specific behaviors a student exhibits within an assessment (Bergner & A. von Davier, 2018). Conversely, biometric data (e.g., eye tracking data, motion tracking data, and neural data) can be used to provide feedback on how the respondents are interacting with the assessment environment, and perhaps most importantly, whether they are motivated and engaged (Agard & A. von Davier, 2018).…”

Robust Estimation for Response Time Modeling

“…Robust estimation for models beyond those applied to item scores collected during the measurement process has not been previously investigated, which is the aim of the current study. With the advent of technology‐enhanced assessment platforms, new information about individuals, such as process data, is becoming more ubiquitous (Bergner & von Daiver, 2018). Data such as eye‐ and motion‐tracking, log data, and response times generated under computerized assessments and game‐based learning systems are gaining unprecedented attention in the field of psychometrics.…”