Reliability as Argument

Abstract

Reliability consists of both important social and scientific values and methods for evidencing those values, though in practice methods are often conflated with the values. With the two distinctly understood, a reliability argument can be made that articulates the particular reliability values most relevant to the particular measurement situation and then the most appropriate evidence and theory to support an argument for the presence of that value. The steps in making a reliability argument are explained and an extended example is given. The article is intended to provoke discussion, debate, and the development of additional reliability methodologies.

Teachers’ approaches to classroom assessment: a large-scale survey

“…Their review further signaled shifts in how measurement theory has been conceptualized for large-scale psychometric uses and classroom teaching and learning contexts. This trend has been articulated by others (Brookhart, 2003;Moss, 2003;Parkes, 2007) who have worked to appropriate measurement concepts for use within classroom settings recognizing that some traditional conceptualizations of validity, reliability, and fairness might not work well for teachers and students. Nonetheless, traditional thinking related to these principles continues to operate and drive classroom assessment practices for some teachers.…”

Chapter 5. Chasing Transparency: Using Disparate Impact Analysis to Assess the (In)Accessibility of Dual Enrollment Composition

“…In that way, assessment-as-normativity-check will differentially impact students/writers from socio-historical situations that are already closer to or further from those dominant norms-a clear zero-sum game in that those born into circumstances closer to the expected norms are privileged to the extent that those further from the expected norms are held down. What is more, while some may suggest that the privileging of proximity to predetermined norms is unavoidable, arguing that there is no assurance of fairness outside of such a move, we note that more and more scholarship over the past quarter century-in both writing assessment (e.g., Gallagher, , 2014O'Neill, 2011;Whithaus, 2005) and educational measurement literature (e.g., Miselvy, 2004;Moss, 1994;Parkes, 2007)-has worked to problematize this premise, suggesting that such methodologies have the type of avoidability that Galtung (1969) noted as a facet of structural violence.…”

On the reliability of vocational workplace-based certifications

“…A number of recent studies have proposed different approaches to reliability analysis that take into account contextual factors as well as assessment scores for reviewing the quality of an assessment result (e.g. Vleuten and Schuwirth 2005;Parkes 2007) or requires raters to make holistic decisions about which of two pieces of students' work is 'better' against an agreed criterion (Pollitt 2012). These models highlight the importance of other quality principles that, although essential to any assessment system, may require special attention in the context of WPVAs since validity and reliability are harder to differentiate.…”