Leveling L2 Texts Through Readability: Combining Multilevel Linguistic Features with the CEFR

Abstract

Selecting appropriate texts for L2 (second/foreign language) learners is an important approach to enhancing motivation and, by extension, learning. There is currently no tool for classifying foreign language texts according to a language proficiency framework, which makes it difficult for students and educators to determine the precise difficulty/complexity levels of an unclassified text. Taking the Chinese language as an example, this study aimed to create a readability assessment system, called the Chinese Readability Index Explorer for Chinese as a Foreign Language (CRIE-CFL), in order to level-that is, to sort by proficiency level-texts that will be used for instructional purposes. The framework of choice in this project is the Common European Framework of Reference (CEFR). A team of expert CFL teachers first classified 1,578 CFL texts into their appropriate CEFR levels. A set of 30 CFL readability features was then developed or drawn from previous research, and sorted according to importance using F-scores. In addition, a support vector machine model was trained by sequentially integrating the features into the model to optimize accuracy. The empirical evaluation of CRIE-CFL revealed average exact-and adjacent-level accuracies of 74.97% and 99.62%, respectively, for predicting the expert classification of a text. The functionalities of CRIE-CFL are introduced and discussed.

Syntactic Complexity in Adapted Teaching Materials: Differences Among Grade Levels and Implications for Benchmarking

“…Sung et al. () had five experienced teachers classify 1,578 texts sampled from representative textbooks into six levels in the Common European Framework of Reference for Languages (CEFR) in terms of text complexity; they then analyzed the texts using 30 linguistic features representative of lexical, syntactic, semantic, and cohesion dimensions with a support vector machine model to determine how well they could predict the CEFR levels of the texts. Their results showed that all individual features had significant predictive power; that the best accuracy was achieved by a model with 25 of the 30 features; and that, strikingly, three lexical complexity features alone were able to achieve an accuracy of 65%.…”

“…Vajjala and Meurers (2012) evaluated the effectiveness of 19 lexical features, 25 syntactic complexity measures, and 2 traditional text readability measures for predicting the grade level of reading texts, and reported an accuracy of 86.7%, 75.3%, and 38.8%, respectively, for each of the three types, and an overall accuracy of 93.3% with all features combined. Sung et al (2015) had five experienced teachers classify 1,578 texts sampled from representative textbooks into six levels in the Common European Framework of Reference for Languages (CEFR) in terms of text complexity; they then analyzed the texts using 30 linguistic features representative of lexical, syntactic, semantic, and cohesion dimensions with a support vector machine model to determine how well they could predict the CEFR levels of the texts. Their results showed that all individual features had significant predictive power; that the best accuracy was achieved by a model with 25 of the 30 features; and that, strikingly, three lexical complexity features alone were able to achieve an accuracy of 65%.…”

A Data‐Driven Approach to Text Adaptation in Teaching Material Preparation: Design, Implementation, and Teacher Professional Development

“…The annotations and benchmark ranges incorporated in the online text adaptation tool are easily manageable by teachers but reflect limited dimensions of the text complexity construct (Crossley et al., ; Sung et al., ). Additional types of annotations and benchmark ranges can be explored in the future; for example, for syntactic complexity (Lu, ).…”

“…A critical consideration in this process that directly bears upon learners’ literacy development is to ensure the linguistic complexity of the reading texts is suitable for their expected proficiency levels (Crossley, Allen, & McNamara, ; Hiebert & Mesmer, ). Matching texts to particular proficiency levels is challenging even for experienced teachers (Sung, Lin, Dyson, Chang, & Chen, ). With the advent of corpus techniques, numerous data‐driven indices are now available for characterizing text complexity (Crossley, Greenfield, & McNamara, ; Graesser, McNamara, & Kulikowich, ; Lu, ).…”

What does corpus linguistics have to offer to language assessment?

“…Most of corpus linguistics research has concentrated on analyzing lexical and syntactical features in written texts in order to understand the complexity of edited texts (Sung, Lin, Dyson, Chang, & Chen, 2015), characteristics of language use in specific domains (Biber et al, 2004), or quality of learner texts in terms of lexical diversity, sophistication, and accuracy, or syntactical complexity, range and accuracy. Correspondingly, these features have attracted the most effort in the machine analysis of linguistic features.…”