Reliability of repeated grading of essay type examinations.

Abstract

The wide variability shown by different teachers in grading the same examinations of the essay type is well known. The classic experiments of Starch and Elliott on reliability of grading history, English, and geometry papers are well known and have been repeated in various forms. 1 But there seems to have been little investigation of the reliability of regrading of the same material by the same teachers. Starch reports a brief investigation in which seven college instructors were asked to regrade a set of ten of their own papers after intervals of two weeks to four years. In this case each question was regraded only once. 2 The object of this paper is to report the results of an experiment in regrading the same set of material after an interval of eleven weeks by sixty-one different teachers, members of the author's summer quarter class in tests and measurements. Practically all were experienced teachers. During the first week of the course they were given mimeographed sheets containing answers from grammer school geography and history papers, with instructions for grading them. The graded papers were handed in the next day. The geography questions were taken from Ruch's experiment; 3 those in history from an experiment reported by Paulu. 4 The questions used and instructions^given were as follows: GEOGRAPHY Name and locate five of the largest cities of the United Stales and name their leading industries, exports, and imports. ANSWER ONEFive of the largest cities in the United States ia Detroit*. An export is Cars. And industry is Manufactoring. Chicago is an important city and an export is SUMMARY Repeated grading of the same essay type of material by the same teachers (sixty-one) after an interval of eleven weeks is very unreliable. Reliability coefficients vary from 0.25 to 0.51. Variability of human judgment in the same individual is about the same as variability between different individuals.

Real‐Time Feedback on Rater Drift in Constructed‐Response Items: An Example From the Golden State Examination

“…In line with earlier research on the statewide tests administered by the CDE (Wilson & Case, 2000;Wilson & Wang, 1995) and research by others from as early as the 1930s (e.g., Ashbum, 1938;Eells, 1930) significant variability between raters was found in their essay scoring. Moreover, significant variation was also found within raters overqtime, confirming findings reported by Braun (1988).…”

A Century of Grading Research

“…Similarly, Jacoby (1910) interpreted his high agreement as a result of the high quality of the papers in his sample. Eells (1930), however, found greater grading consistency in the poorer papers. Lauterbach (1928) found more grading variability for typewritten compositions than for handwritten versions of the same work.…”

“…Many of these early studies communicated a "what's wrong with teachers" undertone that today would likely be seen as researcher bias. Early researchers attributed sources of variation in teachers' grades to one or more of the following sources: criteria (Ashbaugh, 1924;Brimi, 2011;Healy, 1935;Silberstein, 1922;Sims, 1933, Starch, 1915Starch & Elliott, 1913a,b), students' work quality (Bolton, 1927;Healy, 1935;Jacoby, 1910;Lauterbach, 1928;Shriner, 1930;Sims, 1933), teacher severity/leniency (Shriner, 1930;Silberstein, 1922;Sims, 1933;Starch, 1915;Starch & Elliott, 1913b), task (Silberstein, 1922;Starch & Elliott, 1913a), scale (Ashbaugh, 1924;Sims, 1933;Starch 1913Starch , 1915, and teacher error (Brimi, 2011;Eells, 1930;Hulten, 1925;Lauterbach, 1928, Silberstein, 1922Starch & Elliott, 1912, 1913a. Starch (1913, Starch & Elliott 1913b found that teacher error and emphasizing different criteria were the two largest sources of variation.…”

Is unreliability in peer review harmful?

“…The difference vis-a-vis peer review is the use of a marking scheme, however imprecise, which is practised by the examiners. This difference is immediately apparent in comparison with university examinations (Byrne 1980;Cox 1967;Eells 1930;Hartog et al 1936;Laming 1990; again excluding mathematical and physical sciences), which usually have no such scheme. Take away any pretence at a marking scheme and the reliability of examination marks falls to near the levels reported for peer review.…”