The effect of rater variables in the development of an occupation-specific language performance test

Abstract

There has been little research into the effect of raters' background on assessments made in language tests. This article explores the effect that the rater's occupational and linguistic background has on assessments made in an occupation-specific oral language test, the Japanese Language Test for Tour Guides. Assessments of 51 test candidates made by 33 such assessors, including native and near-native speakers of Japanese with backgrounds either in teaching Japanese as a foreign language or in tour guiding in Japanese, were compared in order to determine what effect background has on assessments made on both linguistic and 'real-world' criteria. Multifaceted Rasch analysis was used, as this allows a range of facets to be modelled; for the purposes of this research we were interested in the facet 'raters', and in particular how the scale was applied for each of the assessment categories. These were then compared across the different types of rater. While differences were found in the assessments according to rater type, these were minor and did not point to the unsuitability of any group as a whole; there were no significant differences between the different types of rater in terms of the overall grade awarded. However, there were significant differences in ratings awarded for some individual criteria, and the fact that these reveal different perceptions of what constitutes good performances has obvious implications not only for the selection and training of raters for occupation- specific language performance tests but also for the development of assess ment procedures.

The Diversity in an English Oral Proficiency Test

“…On the other hand, other variables such as occupation, gender, and rater training did not influence test scores [7,15,19]. This implies that there might be unexamined potential variables of individual raters that influence test groups in their rating of L2 oral tests [12,16].…”

“…There are many researchers in the area of oral proficiency tests [2,6,7,9,14,20,23,24]. The importance of the effects of test-taker characteristics on candidate performance in tests of language proficiency has emphasized [14].…”

“…There have been some previous studies of the relationship between raters and test scores of L2 oral performance assessments [5,7,15,17,23,25]. All these studies have found significant differences among raters.…”

Language for Specific Purposes

“…It has the dual purpose of indicating to employers the language proficiency of applicants for positions as Japanese-speaking tour guides, and as a selection criterion for applicants to tour-guide training courses (Brown, 1995). The 30-minute test, consisting of five role plays, was produced in consultation with experienced tour guides, and the assessment criteria were based on their judgements about the necessary features of quality tour-guide communication, referred to as 'task fulfilment criteria' (Brown, 1993), which include such features as enthusiasm, empathy, making something sound interesting, persuasiveness, and awareness of interlocutors' needs or desires.…”

Factors Affecting Item Difficulty in English Listening Comprehension Tests

“…Brown [9] argued that cognitive load is one of the most crucial determinants of difficulty in listening. Cognitive factors analyzed and used in tests can be employed for predicting or adjusting item difficulty as well as providing the evidence of constructs validity.…”