Applications of the Multifaceted Rasch Model

Wolfe, Edward W.; Dobria, Lidia

doi:10.4135/9781412995627.d7

Cited by 10 publications

(8 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This means that, on average, the two most extreme raters assigned ratings that were 0.9 raw score points apart when we compare their mean ratings, and 1.13 raw score points apart when we compare their fair averages. We could report either value as the spread of rater severity, but reporting the fair average spread is warranted when interpreting results from assessment settings in which not all raters rated all tasks (Wolfe & Dobria, 2008). When an observed fair average spread is greater than one (or one level of proficiency on a 4-point scale), then that indicates substantive differences between the most lenient and the most severe raters in their uses of the categories on the rating scale.…”

Section: Resultsmentioning

confidence: 99%

Raters’ L2 background as a potential source of bias in rating oral performance

Winke¹,

2012

View full text Add to dashboard Cite

Based on evidence that listeners may favor certain foreign accents over others (Gass & Varonis, 1984; Major, Fitzmaurice, Bunta, & Balasubramanian, 2002; Tauroza & Luk, 1997) and that language-test raters may better comprehend and/or rate the speech of test takers whose native languages (L1s) are more familiar on some level (Carey, Mannell, & Dunn, 2011; Fayer & Krasinski, 1987; Scales, Wennerstrom, Richard, & Wu, 2006), we investigated whether accent familiarity (defined as having learned the test takers’ L1) leads to rater bias. We examined 107 raters’ ratings on 432 TOEFL iBTTM speech samples from 72 test takers. The raters of interest were L2 speakers of Spanish, Chinese, or Korean, while the test takers comprised three native-speaker groups (24 each) of Spanish, Chinese, and Korean. We analyzed the ratings using a multifaceted Rasch measurement approach. Results indicated that L2 Spanish raters were significantly more lenient with L1 Spanish test takers, as were L2 Chinese raters with L1 Chinese test takers. We conclude by concurring with Xi and Mollaun (2009, 2011) and Carey et al. that rater training should address raters’ linguistic background as a potential rater effect. Furthermore, we discuss the importance of recognizing rater L2 as a possible source of bias.

show abstract

Section: Resultsmentioning

confidence: 99%

Raters’ L2 background as a potential source of bias in rating oral performance

Winke¹,

2012

View full text Add to dashboard Cite

show abstract

“…This means that, on average, the two most extreme MSU raters assigned ratings that were 0.9 raw score points apart when we compare their mean ratings, and 1.13 raw score points apart when we compare their fair averages. We could report either as the spread of rater severity, but reporting the fair average spread is particularly warranted when interpreting our results because not all raters rated all tasks (Wolfe & Dobria, 2008). With a fair average spread > 1 (or one level of proficiency on a 4-point scale), we can see that the range of rater severity is fairly wide.…”

Section: Figure 2 Variable Map From the Facets Analysis Of The Datamentioning

confidence: 95%

The Relationship Between Raters' Prior Language Study and the Evaluation of Foreign Language Speech Samples

Winke

Gass

Myford

2011

ETS Research Report Series

View full text Add to dashboard Cite

This study investigated whether raters' second language (L2) background and the first language (L1) of test takers taking the TOEFL iBT® Speaking test were related through scoring. After an initial 4‐hour training period, a group of 107 raters (mostly of learners of Chinese, Korean, and Spanish), listened to a selection of 432 speech samples that 72 test takers (native speakers of Chinese, Korean, and Spanish) produced. We analyzed the rating data using a multifaceted Rasch measurement approach to uncover potential biases in the rating process. In addition, 26 of the raters participated in stimulated recall sessions, during which they watched videos of themselves rating. Using the video as a prompt, we asked them to discuss and explain their rating processes at the time of rating. The results from our bias interaction analyses revealed that matches between the raters' L2 and the test takers' L1 resulted in some of the raters assigning ratings that were significantly higher than expected. As a whole, raters with Spanish as an L2 were significantly more lenient toward test takers who had Spanish as an L1, and raters with Chinese as an L2 were significantly more lenient toward test takers who had Chinese as an L1. Analyses of the qualitative data, assisted by the program QSR NVivo 8, revealed information concerning the raters' awareness of their biases.

show abstract

“…For each element of each facet, the analysis provides a measure in log‐odds (logit) units, a standard error (information about the precision of that measure), and fit indices (information about how well the data fit the measurement model's expectations). Descriptions of how to interpret these indices are provided elsewhere (Wolfe & Dobria, 2008).…”

Section: Multifaceted Rasch Measurement As An Approach For Detecting mentioning

confidence: 99%

Monitoring Rater Performance Over Time: A Framework for Detecting Differential Accuracy and Differential Scale Category Use

Myford

Wolfe

2009

J Educational Measurement

Self Cite

View full text Add to dashboard Cite

In this study, we describe a framework for monitoring rater performance over time. We present several statistical indices to identify raters whose standards drift and explain how to use those indices operationally. To illustrate the use of the framework, we analyzed rating data from the 2002 Advanced Placement English Literature and Composition examination, employing a multifaceted Rasch approach to determine whether raters exhibited evidence of two types of differential rater functioning over time (i.e., changes in levels of accuracy or scale category use). Some raters showed statistically significant changes in their levels of accuracy as the scoring progressed, while other raters displayed evidence of differential scale category use over time.

show abstract

Applications of the Multifaceted Rasch Model

Cited by 10 publications

References 0 publications

Raters’ L2 background as a potential source of bias in rating oral performance

Raters’ L2 background as a potential source of bias in rating oral performance

The Relationship Between Raters' Prior Language Study and the Evaluation of Foreign Language Speech Samples

Monitoring Rater Performance Over Time: A Framework for Detecting Differential Accuracy and Differential Scale Category Use

Contact Info

Product

Resources

About