Measuring Essay Assessment: Intra-rater and Inter-rater Reliability

Abstract

Problem Statement: There have been many attempts to research the effective assessment of writing ability, and many proposals for how this might be done. In this sense, rater reliability plays a crucial role for making vital decisions about testees in different turning points of both educational and professional life. Intra-rater and inter-rater reliability of essay assessments made by using different assessing tools should also be discussed with the assessment processes. Purpose of Study:The purpose of the study is to reveal possible variation or consistency in grading essay writing ability of EFL writers by the same/different raters using general impression marking (GIM), essay criteria checklist (ECC), and essay assessment scale (ESAS), and discuss rater reliability.Methods: Quantitative and qualitative data were used to present the discussion and implications for the reliability of ratings and the consistency of the measurement results. The assessing tools were applied to 44 EFL university students and 10 graders assessed the essay writing ability of the students by using GIM, ECC, and ESAS in different occasions. Findings and Results:The findings and results of the analyses indicated that using general impression marking is evidently not reliable for assessing essays. The coefficients obtained from checklist and scale assessments, considering the correlation coefficients, estimated variance components, and generalizability coefficients present valuable information, clearly show that there is always variation among the results.

The Effect of Enamel Pre-treatment with Silver Diamine Fluoride versus Nano Silver Fluoride on the Microleakage of Fissure Sealant: in vitro study.

“…Each section was examined for dye penetration. The micro-leakage score was given by a single examiner who was blinded to the treatment and who was trained according to intra-rater agreement (31) to the following scoring system (32) : 0= no evidence of dye penetration; 1= dye penetration of less than 1/3 from the margin of restoration; 2= dye penetration of more than 1/3 and less than 2/3 from the margin of restoration; 3= dye penetration of more than 2/3 from the margin of restoration (Figure 1).…”

The validity of the Spelling and Grammar Waiver as a reasonable accommodation in Leaving Certificate examinations in Ireland

“…In fact, in a significant proportion of cases, scripts marked with a SGW received lower marks than the identical script marked in the standard fashion. While this may be due to the unreliability of essay marking (Kayapınar, 2014), it is also possible that the slightly different marking system for students with a SGW may have a negative impact or that examiners may be biased against some students (for example those who write well with few spelling errors and yet are claiming a waiver for spelling and grammar). Whatever the reason, findings like these are important because they contradict the common assumption that the SGW will benefit all those who receive it.…”

The Relationship Between Linguistic Ability, Multilingualism, and Dementia

“…For instance, why do the authors not report any intra-rater [3] or interrater reliability scores [4]? Previous research has shown that the reliability of scoring written material (regardless of the rating method used) is often low [5]. Was there a specific reason the idea density and the grammatical complexity scores were based on only the final ten sentences of each autobiography?…”