A many-facet Rasch analysis of the second language group oral discussion task

Abstract

FACETS many-facet Rasch analysis software (Linacre, 1998a) was utilized to look at two consecutive administrations of a large-scale (more than 1000 examinees) second language oral assessment in the form of a peer group discussion task with Japanese English-major university students. Facets modeled in the analysis were examinee, prompt, rater, and ve rating category 'items.' Unidimensionality was shown to be strong in both datasets, and approaches to interpreting t values for the facets modeled in the analysis were discussed. Examinee ability was the most substantial facet, followed by rater severity, and item. The prompt facet was negligible in magnitude. Rater differences in terms of severity were generally large, but this characteristic was not stable over time for individuals; returning raters tended to move toward greater severity and consistency, while new raters showed much more inconsistency. Analysis of the scales showed general validity in gradations of scale steps, though raters had some dif culty discerning between categories at the ends of the scales for pronunciation and communicative skills.

New and not so new methods for assessing oral communication

“…For example, in the group oral placement test described in Bonk and Ockey (2003), test takers' performances are rated on pronunciation, fluency, grammar, vocabulary/content, and communicative skills/strategies, the latter being essentially another name for interactional competence. In the group discussion task of the CET-SET, the evaluative criteria include: 1) accuracy in pronunciation, stress/intonation, and use of grammar and vocabulary, 2) range of vocabulary and grammatical structures, 3) size (percentage) of contribution to group discussion, 4) discourse management, 5) flexibility in dealing with different situations and topics, and 6) appropriateness in the use of linguistic resources (Zheng and Cheng, 2008).…”

Predictive modeling of human placement decisions in an English Writing Placement Test

“…In all, the MFRM results suggested that rater leniency or severity was not stable over time. The findings of the current study were in line with Bonk and Ockey's (2003) findings that rater severity may change over time for individuals.…”

“…Several language testing studies (e.g., Bonk & Ockey, 2003;North & Jones, 2009;O'Sullivan, 2008) have employed MFRM approaches in illuminating rater severity effects in assessment processes. The underlying conceptualization of MFRM is that raters are independent readers who subjectively apply their understanding in their ratings and are not considered as 'human clones' to automatically use scoring rubric (Bond & Fox, 2007, p. 114).…”

“…APPENDIX F contains a sample of a data file that contained 1,982 ratings of 845 examinees evaluated by 46 raters. A piece of FACETS codes for the MFRM analysis of this project were adapted from a study by Bonk and Ockey (2003).…”

Assessing speaking ability in academic context: Focusing on a mixed methods approach

“…Others appear to be more skillful in handling dialogue type interview tests. Still others perform best in discussion activities (Brown 2003;Bonk & Ockey, 2003).…”