Rubrics, prototypes, and exemplars: Categorization theory and systems of writing placement

Testing the Test: Expanding the Dialogue on Technical Writing Assessment in the Academy and Workplace

“…This problem is especially evident in rubrics that have multiple categories. Richard Haswell (1998) explains this conundrum well:My experience has always been that it is nearly impossible to find anchor essays that are true to the scale of quality pictured by the rubric. What is pictured is quality rising ladder-like and unilaterally across all subskills.…”

“…In addition, if enough raters score a particular text, eventually the text will receive an average score that appears to show agreement. However, the minimum and maximum scores may vary significantly from the average, which raises the question of which scores should be considered to be the “true score” (Haswell, 1998, p. 242). Disagreement may occur because a text does not conform to the expectations of a rubric, causing raters to waffle about the scores they ought to give.…”

“…Holistic scoring assumes that a score derived from a rubric measures a writer’s ability, when there is actually more evidence that it measures performance at a single task. In high-stakes assessment, such as the writing tests given to employees by potential employers, the goal is not just to assess the text, but to place the writer herself into a category (Haswell, 1998, p. 244). Although classical test theory argues that the text reflects the writer’s abilities, Elbow (2006) argues that an assessor must look “through the text to try to see the writing abilities of the writer behind it” (p. 83).…”

“…The writing tests given to new hires serve a parallel purpose to academic placement exams, in that they are a high-stress, high-risk situation that aims to evaluate writer ability rather than the quality of the completed task (Haswell, 1998; Moss, 1994; Perney, 1994). However, while academic assessment measures ability with the aim to improve the students’ learning, workplace assessment is driven by market forces and is seen in terms of return on investment.…”

A comparability study on differences between scores of handwritten and typed responses on a large-scale writing assessment

“…It also provides better control over the type of writing used, as well as the time spent composing a response, which allows for more equitable comparisons among examinee responses (NAEP, 2011). Standardized writing assessments require objective scoring procedures by trained raters, which adds credibility to scores that may otherwise be questionable if responses were scored by a school's own faculty (Broad, 2003;Haswell, 1998). Having valid, reliable, and comparable scores is critical given the high-stakes nature of the decisions that are often (at least partially) based on exams that incorporate direct writing.…”