Guidelines for the Validation of Writing Assessment in Intervention Studies

Abstract

To draw valid conclusions about the effectiveness of writing interventions, the assessment design is key. Within a writing assessment, observations of students' performance on a particular writing task are generally used to make inferences about their writing proficiency. To do that, researchers have to make various decisions, e.g., about the kind of text(s) students have to write, how the texts are scored and by whom. Moreover, they have to underpin these decisions with arguments and evidence to make sure the conclusions they draw based upon text quality scores are valid interpretations of students' writing proficiency. This requires a thorough and comprehensive analysis of theoretical, empirical, and analytical evidence of writing assessment information, which is often not manageable for researchers whose primary focus is on the writing intervention (Shaw et al., 2012).The aim of this chapter is to provide specific guidelines for researchers on how to make evidence-based choices when assessing writing in a writing intervention study. As such, the guidelines are meant to empower researchers to argue and support the validity of their decisions in and about the assessment of writing. To do that, we will use an existing framework for an argument-based approach to validity (Kane, 2006; and apply it to writing by reviewing theoretical and empirical evidence from writing assessment research.The chapter starts with a brief discussion of Kane's framework that presents a chain of inferences which researchers need to make on the basis of assessment scores. An in-depth discussion of Kane's framework remains outside this chapter's scope, but we will use it to specify the inferences that are generally made within the context of a writing intervention study and the kind of evidence that is needed to substantiate those inferences. For each proposed inference, we will use insights from writing assessment research to provide evidence-based guidelines for researchers to carefully select the most appropriate assessment procedures and to build their own case for the validity of these inferences.

Writing Intervention Research in Authentic Classroom Contexts: A Brief Introduction

“…Writing researchers have to make many choices when designing a writing assessment, all of which may impact a valid interpretation and use of the assessment scores. In chapter 10, Bouwer, Van Steendam, and Lesterhuis (2023) provide support to writing researchers in thinking through the inferences and assumptions underlying these choices and the extent to which they can ultimately claim a valid interpretation of the effectiveness of the intervention on the basis of students' writing assessment performance.…”