Facilitating the interpretation of English language proficiency scores: Combining scale anchoring and test score mapping methodologies

Abstract

The aim of this study was to develop, for the benefit of both test takers and test score users, enhanced TOEFL ITP ® test score reports that go beyond the simple numerical scores that are currently reported. To do so, we applied traditional scale anchoring (proficiency scaling) to item difficulty data in order to develop performance descriptors for multiple levels of each of the three sections of the TOEFL ITP. A (novel) constraint was that these levels should correspond to those established in an earlier study that mapped (i.e., aligned) TOEFL ITP scores to a widely accepted framework for describing language proficiency-the Common European Framework of Reference (CEFR). The data used in the present study came from administrations of five current operational forms of the recently revised TOEFL ITP test. The outcome of the effort is a set of performance descriptors for each of several levels of TOEFL ITP scores for each of the three sections of the test. The contribution, we believe, constitutes (1) an enhancement of the interpretation of scores for one widely used assessment of English language proficiency and (2) a modest contribution to the literature on developing proficiency descriptors-an approach that combines elements of both scale anchoring and test score mapping.

Developing individualized feedback for listening assessment: Combining standard setting and cognitive diagnostic assessment approaches

“…Some maintain that standard setting is embedded in linking, which comprises five procedures including familiarization, specification, standardization training, standard setting, and validation (Council of Europe, 2009). We concur with the latter view that linking is a broader concept encompassing standard setting in that level descriptors in linking have meaning outside the assessment while those in standard setting are generally confined to the assessment within a certain context (Kenyon, 2012, Powers et al, 2017).…”

“…Although combining standard setting and CDA approaches presents a plausible way to provide individualized descriptions about each test taker’s ability to process certain levels of written and oral texts (Green, 2018; Powers et al, 2017) and their cognitive strengths and weaknesses while processing (Jang et al, 2015; Kim, 2015), to our knowledge, there has not been any effort in combining the two approaches to provide individualized feedback for test takers in order to facilitate remedial learning and instruction in language assessment research and practice. This is probably owing to the fact that little is known about the relationship between performance-level classification based on standard setting and mastery/non-mastery classification of subskills based on CDA.…”

Evaluating subscore uses across multiple levels: A case of reading and listening subscores for young EFL learners

“…If it is desired to provide more fine-grained feedback at the individual level than the section scores based on the TOEFL Primary Reading and Listening test, it appears prudent to follow the advice of to consider alternatives such as scale anchoring (e.g., Beaton & Allen, 1992) and item mapping (e.g., Zwick, Senturk, Wang, & Loomis, 2001). These alternatives have been successfully applied to standardized language proficiency tests to provide qualitative descriptions of score ranges (e.g., Gomez, Noah, Schedl, Wright, & Yolkurt, 2007;Powers, Schedl, & Papageorgiou, 2017).…”

Examining the Applications and Opinions of the TOEFL ITP® Assessment Series Test Scores in Three Countries

“…An outstanding question is whether users typically set TOEFL ITP cut scores based on the alignment of these scores to the CEFR levels (several interview participants mentioned doing this). Using the CEFR to set cut scores on the TOEFL ITP may not be ideal given (a) that the performance levels and descriptors of the CEFR are intentionally broad and generic so that they can be applied in a variety of educational contexts and (b) the interpretation of the framework and its levels is up to test users (Papageorgiou, Tannenbaum, Bridgeman, & Cho, ; Powers, Schedl, & Papageorgiou, ).…”

“…Respondents did indicate wanting additional information about the TOEFL ITP, such as how test scores relate to scores on other English‐language assessments or to the CEFR. Information about how TOEFL ITP Level 1 scores relate to the CEFR is available on the ETS website or from publications (Powers et al, ; Tannenbaum & Baron, ); corresponding information for TOEFL ITP Level 2 is under development at the time of writing. Some users may be unaware of the existence of this information for Level 1 or, perhaps, have difficulty processing this information in English.…”