Predicting lexical proficiency in language learner texts using computational indices

Abstract

indices Predicting lexical proficiency in language learner texts using computational Published by: http://www.sagepublications.com can be found at: Language Testing Additional services and information for AbstractThe authors present a model of lexical proficiency based on lexical indices related to vocabulary size, depth of lexical knowledge, and accessibility to core lexical items. The lexical indices used in this study come from the computational tool Coh-Metrix and include word length scores, lexical diversity values, word frequency counts, hypernymy values, polysemy values, semantic coreferentiality, word meaningfulness, word concreteness, word imagability, and word familiarity. Human raters evaluated a corpus of 240 written texts using a standardized rubric of lexical proficiency. To ensure a variety of text levels, the corpus comprised 60 texts each from beginning, intermediate, and advanced second language (L2) adult English learners. The L2 texts were collected longitudinally from 10 English learners. In addition, 60 texts from native English speakers were collected. The holistic scores from the trained human raters were then correlated to a variety of lexical indices. The researchers found that lexical diversity, word hypernymy values and content word frequency explain 44% of the variance of the human evaluations of lexical proficiency in the examined writing samples. The findings represent an important step in the development of a model of lexical proficiency that incorporates both vocabulary size and depth of lexical knowledge features.

Assessing writing ability in primary education : on the evaluation of text quality and text complexity

“…Regarding the use of connectives, the present finding that the overall use of cohesive features decreased with grade level is supported by Crossley et al (2011). Thus, although a coherent text indicates high writing ability (Sanders & Schilperoord, 2006;Sanders et al, 1996), coherence seems achieved by the use of relatively fewer cohesive features as writing ability increases.…”

“…For example, the relation between lexical complexity and lexical diversity is supported by results from Crossley et al (2011), who reported the use of increasingly sophisticated words as grade level increased. In a study by McNamara et al (2010), lexical diversity and word frequency were found to be the most predictive features.…”

“…As Crossley et al (2011) Hayes & Flower, 1980;McCutchen & Perfetti, 1982;Perfetti & McCutchen, 1987, Sanders & Van Wijk, 1996Van Wijk & Sanders, 1999). However, the exact relation between linguistic features and text quality has to be determined for each language and population to which AEE is applied.…”

Vocabulary Growth in College-Level Students' Narrative Writing

“…Crossley, et al (2010), for instance, indicated that, "Perhaps the most robust finding of this study is that an index of lexical diversity, D, explains almost 34% of the variance in human judgments of written lexical proficiency" (p.13). This shows that lexical diversity affects human assessors' perceptions of writing quality and therefore could differentiate between different proficiency levels.…”

Comparing the Lexical Features of EAP Students’ Essays by Prompt and Rating

“…Accessibility refers to how quickly words can be retrieved or processed, and is based on judgements of word concreteness and familiarity. Using computational tools that generate lexical profiles, such as Coh-Metrix (Graesser, McNamara, Louwerse, & Cai, 2004), researchers have identified which lexical features differentiate texts written by L1 writers at varying grade levels (Crossley, Weston, McLain Sullivan, & McNamara, 2011;McNamara, Crossley, & McCarthy, 2010) and texts written by L2 writers from diverse proficiency levels (Crossley & McNamara, 2012;Crossley, Salsbury, & McNamara, 2011;Crossley, Salsbury, McNamara, & Jarvis, 2010). The L2 studies showed that lexical diversity and word frequency, both measures of vocabulary size, and word hypernymy (a measure of breadth) had strong relationships with ratings of lexical proficiency, while lexical diversity, word familiarity, frequency, meaningfulness, and imagability (i.e., ease of constructing a mental image) had strong relationships with ratings of text quality.…”