Back to Basics: How Measures of Lexical Diversity Can Help Discriminate between CEFR Levels

Abstract

This study contributes to ongoing discussions on how measures of lexical diversity (LD) can help discriminate between essays from second language learners of English, whose work has been assessed as belonging to levels B1 to C2 of the Common European Framework of 1 We are very grateful to Pearson Education Ltd for sponsoring us with a research grant that made this study possible, and to Kirsten Ackermann, Veronica Benigno and Jeremy Hancock for their support in carrying out this project. 2 This work was funded by Language Testing, a trading division of Pearson Education Ltd.We are very grateful to Kirsten Ackermann, Veronica Benigno and Jeremy Hancock for their support in carrying out this project. We are also indebted to Scott Jarvis for providing a spreadsheet to help us with computing HD-D. Reference (CEFR). The focus is in particular on how different operationalisations of what constitutes a "different word" (type) impact on the LD measures themselves and on their ability to discriminate between CEFR levels. The results show that basic measures of LD, such as the number of different words, the TTR (Templin 1957) and the Index of Guiraud (Guiraud 1954) explain more variance in the CEFR levels than sophisticated measures, such as D (Malvern et al. 2004), HD-D (McCarthy and Jarvis 2007) and MTLD (McCarthy 2005) provided text length is kept constant across texts. A simple count of different words (defined as lemma's and not as word families) was the best predictor of CEFR levels and explained 22 percent of the variance in overall scores on the Pearson Test of English Academic in essays written by 176 test takers.

Word Families and Frequency Bands in Vocabulary Tests: Challenging Conventions

“…Because the members of a lemma differ only in grammatical form rather than lexicosemantic properties (at least in most cases), knowledge of the lemma representative on the test would most likely imply knowledge of the other lemma members. Also, lemmas have already been shown to hold advantages over word families in lexical diversity measurement (Treffers‐Daller, Parslow, & Williams, ). There is evidence for the psycholinguistic reality of lemmas in L1 speakers, which is likely to be similar in L2 learners (see Aitchison, , for an overview), although further research will be required to see if this transfers to a testing environment.…”

Lexical richness of Chinese candidates in the graded oral English examinations

“…Daller, van Hout and Treffers-Daller (2003) also argue that Guiraud, the mathematical transformation of TTR compensates for the systematically falling TTR with increasing text length, but that it is not always clear whether it over-or under compensates. Nevertheless, Guiraud has been used successfully in a recent study by Treffers-Daller, Parslow and Williams (2016), who show that Guiraud and simply the number of different words used by the candidates are the best predictors for language proficiency at different levels of the Common European Framework (Council of Europe,…”

Empirical Learner Language and the Levels of the Common European Framework of Reference

“…It is also highly task‐dependent, so that sometimes text length is chosen as a controlled variable (e.g., Crossley & McNamara, ). In a study focusing on measures for the prediction of the lexical diversity of N = 179 CEFR‐related essays (B1–C2, English as a Second Language), Treffers‐Daller, Parslow, and Williams, , first show that different lemmatization principles considerably effect lexical diversity measures (Treffers‐Daller et al, ). Holding text length constant, the authors then demonstrate that, for their data, “basic” measures (e.g., Guiraud's index, type‐token ratio) highly correlated among each other have more explanatory power than more complex measures such as D, HD‐D, or MTLD.…”