Rethinking validation in complex high-stakes assessment contexts

Abstract

In this article we rethink validation within the complex contexts of high-stakes assessment. We begin by considering the utility of existing models for validation and argue that these models tend to overlook some of the complexities inherent to assessment use, including the multiple interpretations of assessment purposes and the potential interaction of assessment uses. We respond to these limitations by proposing an interpretive approach to validation that we call validation as narrative case description. This approach uses a case-based framework and hermeneutic methodology to construct and analyse validity evidence and suggests narrative as a representational format for communicating validation claims. We illustrate this approach by considering a case study of a secondary mathematics assessment administered in Ontario, Canada. In introducing this approach, we are contributing to the ongoing dialogue on reconceptualising validation as an interpretive process that serves a generative function.

Challenges for information technology supporting educational assessment

“…These findings and others (e.g. see Koch & DeLuca, ; Mansell et al ., ; Stanley, MacCann, Gardner, Reynolds, & Wild, ) suggest that involving teachers in assessment design is an important element of teaching that can contribute to learning. Furthermore, removing this responsibility from teachers through a focus on high‐stakes assessment can be damaging to students' learning (Mansell et al ., ).…”

“…A review of the literature (Koch & DeLuca, ) suggests that multiple uses of assessment is neither a recent phenomenon nor a rare occurrence (p. 3). However, there is general agreement that the purposes of the assessment must be clarified early in the design process in order for an assessment to embody a complete and coherent argument (Mislevy et al ., ) and to be valid (Crooks, Kane, & Cohen, ).…”

Validity in work-based assessment: expanding our horizons

“…Validation can then be defined as ‘developing a scientifically sound validity argument to support the intended interpretation of test scores and their relevance to the proposed use’ through accumulation and integration of different kinds of evidence from different sources. Or, as stated by Koch and DeLuca: ‘..validation should be a generative process that promotes continuous inquiry into assessment practice’. What is rarely addressed explicitly, though, is that our approaches to WBA – reflected in the way we design and evaluate assessment practices – are inextricably linked to our implicit theories of learning, performance and competence.…”

Theoretical Considerations for Game-Based e-Learning Analytics

“…In addition to the technical problems of new dimensions of assessment and new nonlinear challenges for analysis, the targets of assessment are also changing, from basic skills and declarative knowledge toward the integration of knowledge applied to complex open-ended tasks requiring critical thinking, communication, collaboration, and creativity (Bennett, Persky, Weiss, & Jenkins, 2007 ;Koch & DeLuca, 2012 ). This shift toward higher order thinking complicates the performance space with constructs that test the boundaries of the sociological (e.g.…”