Setting cut scores on an EFL placement test using the prototype group method: A receiver operating characteristic (ROC) analysis

Abstract

This paper presents an approach to standard setting that combines the prototype group method (PGM; Eckes, 2012) with a receiver operating characteristic (ROC) analysis. The combined PGM-ROC approach is applied to setting cut scores on a placement test of English as a foreign language (EFL). To implement the PGM, experts first named learners whom they considered to be typical of each of five levels of language proficiency as specified by the Common European Framework of Reference for Languages (CEFR; Council of Europe, 2001). Out of a total of 3,310 examinees taking different trial versions of the placement test, 470 learner prototypes were identified. For this set of prototypes, Rasch model estimates of EFL proficiency served as input to a series of ROC analyses, one for each pair of adjacent proficiency levels. Cut scores were derived using the Youden index that maximizes the overall rate of correct classification and minimizes the overall rate of misclassification. Findings confirmed that this method allows for the setting of cut scores that show a high level of classification accuracy in terms of the correspondence with expert categorizations of examinee prototypes. In addition, the ROC-based cut scores were associated with higher classification accuracy than cut scores derived from a logistic regression analysis of the same data. Potential further uses and implications of the PGM-ROC approach in the context of language testing and assessment are discussed.

Mathematics and science proficiency of young refugees in secondary schools in Germany

“…C-tests are a variant of the cloze principle (Klein-Braley, 1997): In a short coherent text, every second half of every second word is missing; participants have to fi ll in the gaps in a meaningful and linguistically correct manner. C-tests are widely used as indicators of general language profi ciency (e.g., Eckes, 2017). In the study at hand, two diff erent texts were used with 30 gaps each.…”

Standardization of the Gender Assignment and Agreement Assessment in the Greek Language: Preliminary Evidence from Bilingual Greek—Albanian School Age Children

“…Additionally, a Receiver Operating Characteristics (ROC) analysis was run to evaluate the test's discriminatory ability with monolingual and bilingual children (Eckes 2017;Fan et al 2006;Huang and Konold 2014), while for the computation of the cut-off scores, the Youden analysis (Youden 1950) was applied, since it is a robust and effective analysis for the calculation of cut-off scores (Fluss et al 2005; Martínez-Camblor and Pardo-Fernández 2019; Mizuta et al 2022;Perkins and Schisterman 2005). ROC analysis sensitivity and specificity values which range between 80% and 89% are considered fair, while rates above 90% are considered good (Plante and Vance 1994;Rhodes et al 2019).…”

IQB Trends in Student Achievement 2018: A Large-Scale Educational Assessment Study in Germany

“…C-tests are a variant of the cloze principle (Klein-Braley, 1997): In a short coherent text, every second half of every second word is missing; participants have to fill in the gaps in a meaningful and linguistically correct manner. C-tests are widely used as indicators of general language proficiency (e.g., Eckes, 2017). In the study at hand, two different texts were used with 30 gaps each.…”