What Are We Aligning Tests to When We Report Test Alignment to the CEFR?

Harsch, Claudia; Hartig, Johannes

doi:10.1080/15434303.2015.1092545

Cited by 12 publications

(6 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Third, owing to its parametric estimation procedure the BLR analysis demands relatively large sample sizes. This may pose real challenges for applying the PGM in contexts where such large samples are difficult to obtain (for a similar point, see Harsch & Hartig, 2015).…”

Section: Summary and Discussionmentioning

confidence: 99%

“…Fourth, the amount of unwanted variability of judgments is increased further by between-judge differences in the severity or leniency exhibited when applying performance standards or criteria (Engelhard, 2009, 2011; Longford, 1996; Van Nijlen & Janssen, 2008). Finally, when aligning tests to a framework that specifies sets of performance categories or proficiency levels, such as the Common European Framework of Reference for Languages (CEFR; Council of Europe, 2001), judges tend to use widely different judgmental strategies and are influenced by factors irrelevant to the judgment task (Harsch & Hartig, 2015; Papageorgiou, 2010; see also Deunk, van Kuijk, & Bosker, 2014).…”

Section: The Critical Role Of Judgments In Standard Settingmentioning

confidence: 99%

See 1 more Smart Citation

Setting cut scores on an EFL placement test using the prototype group method: A receiver operating characteristic (ROC) analysis

Eckes¹

2016

Language Testing

View full text Add to dashboard Cite

This paper presents an approach to standard setting that combines the prototype group method (PGM; Eckes, 2012) with a receiver operating characteristic (ROC) analysis. The combined PGM-ROC approach is applied to setting cut scores on a placement test of English as a foreign language (EFL). To implement the PGM, experts first named learners whom they considered to be typical of each of five levels of language proficiency as specified by the Common European Framework of Reference for Languages (CEFR; Council of Europe, 2001). Out of a total of 3,310 examinees taking different trial versions of the placement test, 470 learner prototypes were identified. For this set of prototypes, Rasch model estimates of EFL proficiency served as input to a series of ROC analyses, one for each pair of adjacent proficiency levels. Cut scores were derived using the Youden index that maximizes the overall rate of correct classification and minimizes the overall rate of misclassification. Findings confirmed that this method allows for the setting of cut scores that show a high level of classification accuracy in terms of the correspondence with expert categorizations of examinee prototypes. In addition, the ROC-based cut scores were associated with higher classification accuracy than cut scores derived from a logistic regression analysis of the same data. Potential further uses and implications of the PGM-ROC approach in the context of language testing and assessment are discussed.

show abstract

Section: Summary and Discussionmentioning

confidence: 99%

Section: The Critical Role Of Judgments In Standard Settingmentioning

confidence: 99%

Setting cut scores on an EFL placement test using the prototype group method: A receiver operating characteristic (ROC) analysis

Eckes¹

2016

Language Testing

View full text Add to dashboard Cite

show abstract

“…Puhuminen on tilannesidonnainen ja sosiaalinen taito, jonka arviointi on monimutkaista. Arvioijien yksilölliset mieltymykset ja tulkinnat vaikuttavat esimerkiksi taitotason määrittämiseen (Harsch & Hartig 2015). Arviointi ei ole yhteismitallista, koska arviointia tekevät saattavat kiinnittää huomiota kielen eri piirteisiin, minkä lisäksi heidän taustansa tai käsityksensä puhumisen taidosta saattavat olla erilaiset (Han 2022).…”

Section: Suullisen Kielitaidon Arvioinnin Lähtökohtiaunclassified

Ihmisarvioijien näkemyksiä suullisen kielitaidon automaattisesta arvioinnista, digitaalisesta arviointiprosessista sekä puhesuorituksista arvioitavista ulottuvuuksista

Zansen

Kallio²,

Sneck

et al. 2022

afinlavk

View full text Add to dashboard Cite

This study investigated human raters’ perceptions of automated assessment of oral language skills. The raters (n = 37) participated in three assessment rounds organized by the DigiTala research project using Moodle and Zoom. The raters assessed Finnish and Swedish learners’ speech samples using one holistic and five analytical rating scales created in the project. After the assessment, the raters responded to a questionnaire that included Likert-scale and open-ended questions. Numerical responses were analyzed with descriptive statistics, open responses with content analysis. The raters think that automated scoring could support human rating. The assessment rounds were carried out successfully. The selected dimensions proved to be essential parts of the speaking performances. The results will benefit those working on automated assessment and oral language assessment.

show abstract

“…29 (S3): 85 -101 (2021) framework has become significant for language testers and examination boards worldwide as it assists language planners to define language proficiency levels and to decipher them into meaningful language credentials. For many language testers, it has become imperative for their exams to align with CEFR (Gyllstad et al, 2014;Harsch & Hartig, 2015;Nunan, 2014;Taylor & Jones, 2006). The Council of Europe has endeavoured to facilitate this by providing a toolkit of resources, including a draft pilot Manual for relating language examinations to the CEFR and a technical reference supplement (Council of Europe, 2020).…”

Section: Common European Framework Of Reference For Languagesmentioning

confidence: 99%

Mapping IIUM Students’ English Language Writing Proficiency to CEFR

Sufi¹,

Ibrahim²

2021

JSSH

View full text Add to dashboard Cite

The goal of this research was to determine the English language writing skills of the International Islamic University of Malaysia (IIUM) graduating undergraduates based on two linguistic levels; the IIUM English Proficiency Test (EPT) and the Common European Framework of Reference for Languages (CEFR), based on two descriptors: IIUM’s EPT descriptor, and CEFR’s ‘General linguistic range’ descriptor (Council of Europe, 2001, p 110). The EPT results show that the majority (48 %) of the English language writing skills of IIUM undergraduates were in Band 6, while Band 5.5, Band 5 and Band 4, respectively, ranked 31.3%, 4.9% and 0.4%. On the other hand, Band 8, Band 7 and Band 6.5 were attained by 0.2%, 2.4% and 12.2% of undergraduates. A significant proportion of IIUM undergraduates (46.3 %) were found to be at Level B2 + when assessed against the CEFR scale for the ‘General linguistic range.’ At the same time, 2.2%, 12.4%, 33.9 and 5.2% of students were at Level C2, C1, B2, B1 +, respectively. The results also showed that a mere 0.2% was at Level B1. This research also showed that despite some inaccuracies and improper uses, IIUM students could write effectively and understand and use reasonably complicated language, particularly in familiar situations. Based on the CEFR scale for ‘General linguistic range’, the results of this study show that IIUM graduating undergraduates could clearly articulate themselves in their writing.

show abstract

What Are We Aligning Tests to When We Report Test Alignment to the CEFR?

Cited by 12 publications

References 19 publications

Setting cut scores on an EFL placement test using the prototype group method: A receiver operating characteristic (ROC) analysis

Setting cut scores on an EFL placement test using the prototype group method: A receiver operating characteristic (ROC) analysis

Ihmisarvioijien näkemyksiä suullisen kielitaidon automaattisesta arvioinnista, digitaalisesta arviointiprosessista sekä puhesuorituksista arvioitavista ulottuvuuksista

Mapping IIUM Students’ English Language Writing Proficiency to CEFR

Contact Info

Product

Resources

About