Investigating rater/prompt interactions in writing assessment: Quantitative and qualitative approaches

Weigle, Sara Cushing

doi:10.1016/s1075-2935(00)00010-6

Cited by 125 publications

(101 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…However, experienced teachers are also advantageous because they are more likely to have a better sense of assigning coherent scores across examinees based upon their experience. Previous research suggests that raters with little experience tend to score tasks more severely than experienced raters (Shohamy, Gordon, & Kraemer, 1992;Weigle, 1999). However, Myford and Mislevy (1994) and Meyer (2000) found that teaching experience had only a negligible impact on predictions of rater severity.…”

Section: Rubrics In Performance Assessmentsmentioning

confidence: 95%

Crossed Random-Effect Modeling: Examining the Effects of Teacher Experience and Rubric Use in Performance Assessments

Kan

Bulut

2014

EJER

View full text Add to dashboard Cite

Problem Statement: Performance assessments have emerged as an alternative method to measure what a student knows and can do. One of the shortcomings of performance assessments is the subjectivity and inconsistency of raters in scoring. A common criticism of performance assessments is the subjective nature of scoring procedures. The effectiveness of the performance assessment procedure depends highly on the quality and coordination of teacher and rubric. To gain a better understanding of the interaction between teachers and performance assessments, it is crucial to examine the effects of teacher-related factors and how teachers interact with scoring rubrics when grading performance assessments. One of these factors is teachers' work and scoring experience. When grading performance assessments, the experienced teachers may be expected to grade student performances more objectively through their experience in instruction and evaluation than the teachers with less teaching and scoring experience.

show abstract

Section: Rubrics In Performance Assessmentsmentioning

confidence: 95%

Crossed Random-Effect Modeling: Examining the Effects of Teacher Experience and Rubric Use in Performance Assessments

Kan

Bulut

2014

EJER

View full text Add to dashboard Cite

show abstract

“…He suggested monitoring raters as they think aloud so as to identify the problematic aspects of scoring. Verbal protocols have been used in studies of oral performance ratings (e.g., Attali, 2016;Barkaoui, 2011;Cumming, Kantor, & Powers, 2002;Kim, 2011;2015;Sasaki, 2014;Weigle, 1999;Wolfe, 2004). According to Davis (2016), the use of thinkaloud protocols can unfold raters' thoughts in order to identify why and how a rater chooses a certain score.…”

Section: Verbal Protocols In Performance Assessmentmentioning

confidence: 99%

“…Weigle (1999), in a study of raters' verbal protocols in writing assessment, found that some raters announced difficulty expressing their thoughts out-loud and that some raters provided much more protocols than the others. Barkaoui (2011) used the Multifaceted Rasch Measurement (MFRM) and verbal protocols in a writing test and could identify misfitting raters.…”

Section: Merits and Demerits Of The Use Of Verbal Protocolsmentioning

confidence: 99%

“…Besides, research on the use of raters' verbal protocols, although highly essential, is rather rare. Even those very few ones on the application of verbal protocols (e.g., Barkaoui, 2011;Kim, 2011Kim, , 2015Sasaki, 2014;Weigle, 1999;Wolfe, 2004) did not use both qualitative and quantitative analysis models together. The reason for such importance is that it is only through verbalization during the rating process that researchers understand how raters make judgments about the quality of oral discourse, or whether raters display interference when providing protocols.…”

Section: Merits and Demerits Of The Use Of Verbal Protocolsmentioning

confidence: 99%

“…Although according to Weigle (1999), a request for verbal reports may affect the raters' scoring, such elicitation method is necessary to observe the raters' decision-making process. The researcher transcribed the verbally-recorded reports based on Shohamy's (1994) discourse features framework to analyze the produced verbal protocols based on lexical density, rhetorical functions and structures, genre, speech moves, communicative properties, discourse strategies, content and topic of discourse, prosodic/paralinguistic features and contextualizations, type of speech functions, discourse markers, and register for qualitative data analysis to achieve further certainty of raters' change of behavior in various rating groups among pre-, post-and delayed post-training stages.…”

Section: Rating With Verbal Protocol Reportsmentioning

confidence: 99%

See 2 more Smart Citations

Investigating the Effect of the Training Program on Raters’ Oral Performance Assessment: A Mixed-Methods Study on Raters’ Think-Aloud Verbal Protocols

Bijani

Khabiri

2017

IJAL

View full text Add to dashboard Cite

Although the use of verbal protocols is growing in oral assessment, research on the use of raters' verbal protocols is rather rare. Moreover, those few studies did not use a mixed-methods design. Therefore, this study investigated the possible impacts of rater training on novice and experienced raters' application of a specified set of standards in rating. To meet this objective, the study made use of verbal protocols produced by 20 raters who scored 300 test takers' oral performances and analyzed the data both qualitatively and quantitatively. The outcomes demonstrated that through applying the training program, the raters were able to concentrate more on linguistic, discourse, and phonological features; therefore, the extent of their agreement increased specifically among the inexperienced raters. The analysis of verbal protocols also revealed that training how to apply a well-defined rating scale can foster its use for raters both validly and reliably. Various groups of raters approach the task of rating in different ways, which cannot be explored through pure statistical analysis. Thus, think-aloud verbal protocols can shed light on the vague sides of the issue and add to the validity of oral language assessment. Moreover, since the results of this study showed that inexperienced raters can produce protocols of higher quality and quantity in the use of macro and micro strategies to evaluate test takers' performances, there is no evidence based on which decision makers should exclude inexperienced raters solely because of their lack of adequate experience.

show abstract

Multifaceted Rasch Analysis for Test Evaluation

Barkaoui

2013

The Companion to Language Assessment

View full text Add to dashboard Cite

This chapter provides a conceptual introduction to Rasch models and their potential applications in language test evaluation, with a specific focus on the multifaceted Rasch model (MFRM). In the MFRM, factors in the assessment setting (e.g., task, test taker, rater) are called facets. The MFRM uses the ratings that raters assign to test‐taker performance to provide parameter estimates for each facet (e.g., rater severity, task difficulty, test‐taker ability). Each facet is calibrated from the observed scores and all facets are placed on a single common linear scale called a variable or facets map. An example focusing on an English writing test is used to illustrate the main points discussed in the chapter and the applications and advantages of the MFRM. The chapter also discusses some of the main issues in Rasch approaches to test evaluation.

show abstract

Investigating rater/prompt interactions in writing assessment: Quantitative and qualitative approaches

Cited by 125 publications

References 28 publications

Crossed Random-Effect Modeling: Examining the Effects of Teacher Experience and Rubric Use in Performance Assessments

Crossed Random-Effect Modeling: Examining the Effects of Teacher Experience and Rubric Use in Performance Assessments

Investigating the Effect of the Training Program on Raters’ Oral Performance Assessment: A Mixed-Methods Study on Raters’ Think-Aloud Verbal Protocols

Multifaceted Rasch Analysis for Test Evaluation

Contact Info

Product

Resources

About