Limitations of the Common European Framework for developing comparable examinations and tests

Abstract

The Common European Framework of Reference (CEFR) posits six levels of proficiency and defines these largely in relation to empirically derived difficulty estimates based on stakeholder perceptions of what language functions expressed by ‘Can-do’ statements can be successfully performed at each level. Though also containing much valuable information on language proficiency and advice for practitioners, in its present form the CEFR is not sufficiently comprehensive, coherent or transparent for uncritical use in language testing. First, the descriptor scales take insufficient account of how variation in terms of contextual parameters may affect performances by raising or lowering the actual difficulty level of carrying out the target ‘Can-do’ statement. In addition, a test’s theory-based validity - a function of the processing involved in carrying out these ‘Can-do’ statements - must also be addressed by any specification on which a test is based. Failure to explicate such context and theory-based validity parameters - i.e., to comprehensively define the construct to be tested - vitiates current attempts to use the CEFR as the basis for developing comparable test forms within and across languages and levels, and hampers attempts to link separate assessments, particularly through social moderation.

Linking English‐language Test Scores Onto the Common European Framework of Reference: An Application of Standard‐setting Methodology

“…The difficulty of linking test scores to the CEFR should not be underestimated. The CEFR, according to Weir (2005), does not provide sufficient information about how contextual factors affect performance across the levels, or adequately delineate how language develops across the levels in terms of cognitive or meta-cognitive processing. This may lead to difficulties in interpreting differences across the CEFR levels.…”

“…The concerns of Weir (2005) regarding the current state of the CEFR led him to report that "it is not surprising that a number of studies have experienced difficulty attempting to use the CEFR for test development or comparability purposes" (p. 283). But surely the imperfect nature of the CEFR is not the sole cause; any test is also an imperfect representation of its intended construct.…”

“…However, the usefulness of the CEFR should be considered with an understanding of some of its shortcomings. Weir (2005) cautioned that while the CEFR provides valuable information on language proficiency, the level descriptors do not offer sufficient information about how contextual factors affect performance across the levels, which he termed context validity. He also expressed concern that the CEFR does not adequately delineate how language develops across the levels in terms of cognitive or meta-cognitive processing.…”

Exámenes certificativos de ELE referenciados a sistemas externos como el del MCER

“…Es necesario recordar, además, que son varios los autores (Alderson, 2007;Alderson et al, 2004;Fulcher, 2004Fulcher, , 2010Weir, 2005;, entre otros) que dan cuenta de las numerosas deficiencias que presenta la formulación de algunos de los descriptores de las escalas del MCER. También, que los propios autores del Marco reconocen que los descriptores de algunas de las escalas, en especial, las de los niveles C1 y C2 no han sido adecuadamente calibrados (North, 2007).…”

Standardized External Testing in Foreign Languages: Ukraine’s Experience

“…In spite of the CEF being a normative document in the sphere of language learning, teaching, and assessment, it should not be considered as a methodological must (North, 2004). The critical analysis of the document points to some drawbacks, among them theoretical discrepancies, terminological gaps, lack of consistency and others (Fulcher, 2004;Weir, 2005). Thus, the function of the CEF should be regarded as facilitating the test design process but not bounding it within its limits.…”