Developing and Validating Band Levels and Descriptors for Reporting Overall Examinee Performance

Abstract

This study presents the development and empirical validation of score levels and descriptors specifically designed for reporting purposes to provide test takers with more than just a number on a score scale. In the context of a test primarily intended for 11-to 15-year-old students learning English as a second/foreign language, the study examined the number of band levels that could be meaningfully distinguished, the reliability of the classification of students into these band levels, and the development of overall performance descriptors that would provide meaningful information to score users. The performance data from 2,931 students who took the test were used. The band level solution was determined by balancing considerations for the reliability of classification decisions and the desire for the levels to represent meaningful performance differences. To construct meaningful descriptors for the band levels, multiple sources of information were examined, including the scoring rubrics, the characteristics of test items, typical student performance profiles, and the performance of norm groups on the test. The importance of establishing the psychometric quality of band levels and the empirical basis for performance descriptors, as well as the implications for similar efforts, are discussed.

Facilitating the interpretation of English language proficiency scores: Combining scale anchoring and test score mapping methodologies

“…Its use with international assessments of English language proficiency has occurred much more recently. For example, there have been several notable efforts associated with the TOEIC (Liao, 2010) and with several members of the TOEFL family of assessments, namely the TOEFL iBT ® test (Garcia Gomez et al, 2007) and TOEFL Junior ® Comprehensive test (Papageorgiou, Xi, Morgan, & So, 2015). Compared with earlier studies, the latter study included two distinguishing features: first, the authors investigated classification reliability to decide on the number of levels to be reported; and, second, the focus of the study was on the reporting of levels and descriptors for overall test performance, which included consideration of responses to both selected-response and constructed-response items, as opposed to performance on a single, language skill-specific test section, containing selected-response items only.…”

TOEFL Junior® Design Framework

“…The following two steps were followed in developing the overall score levels and level descriptors: (a) developing band levels and (b) developing performance descriptors. More details about the procedures can be found in Papageorgiou, Morgan, and Becker () for the TOEFL Junior Standard test and in Papageorgiou, Xi, Morgan, and So (in press) for the TOEFL Junior Comprehensive test.…”

“…Table shows the relationship of the overall score levels between the tests. The results in the table were produced as part of the study that developed the overall score levels for TOEFL Junior Standard (Papageorgiou et al, ).…”

Can the National Center Test in Japan be replaced by commercially available private English tests of four skills? In the case of TOEFL Junior Comprehensive

“…Because the speaking and writing section scores will not be equated, the scores are not strictly comparable … To avoid any incorrect impression on the part of stakeholders that the speaking and writing scores are comparable across forms, Following this concept, instead of total scaled scores, overall score levels compatible with CEFR (see Tannenbaum and Baron 2015, for details) are reported in JC ranging from 1 to 6 (see Papageorgiou et al 2015, for details). The use of different scoring scales for receptive skills (reading and listening) and productive skills (speaking and writing) imply that at least these two constructs need to be separately posited in the model.…”