Predicting the proficiency level of language learners using lexical indices

Abstract

This study explores how second language (L2) texts written by learners at various proficiency levels can be classified using computational indices that characterize lexical competence. For this study, 100 writing samples taken from 100 L2 learners were analyzed using lexical indices reported by the computational tool Coh-Metrix. The L2 writing samples were categorized into beginning, intermediate, and advanced groupings based on the TOEFL and ACT ESL Compass scores of the writer. A discriminant function analysis was used to predict the level categorization of the texts using lexical indices related to breadth of lexical knowledge (word frequency, lexical diversity), depth of lexical knowledge (hypernymy, polysemy, semantic co-referentiality, and word meaningfulness), and access to core lexical items (word concreteness, familiarity, and imagability). The strongest predictors of an individual's proficiency level were word imagability, word frequency, lexical diversity, and word familiarity. In total, the indices correctly classified 70% of the texts based on proficiency level in both a training and a test set. The authors argue for the applicability of a statistical model as a method to investigate lexical competence across language levels, as a method to assess L2 lexical development, and as a method to classify L2 proficiency.

Comparing the Lexical Features of EAP Students’ Essays by Prompt and Rating

“…The L2 studies showed that lexical diversity and word frequency, both measures of vocabulary size, and word hypernymy (a measure of breadth) had strong relationships with ratings of lexical proficiency, while lexical diversity, word familiarity, frequency, meaningfulness, and imagability (i.e., ease of constructing a mental image) had strong relationships with ratings of text quality. In essence, L2 writers at higher proficiency levels use more imagable, unfamiliar, and infrequent words and show greater lexical diversity than lower proficiency writers (Crossley, Salsbury, et al, 2011).…”

“…Accessibility refers to how quickly words can be retrieved or processed, and is based on judgements of word concreteness and familiarity. Using computational tools that generate lexical profiles, such as Coh-Metrix (Graesser, McNamara, Louwerse, & Cai, 2004), researchers have identified which lexical features differentiate texts written by L1 writers at varying grade levels (Crossley, Weston, McLain Sullivan, & McNamara, 2011;McNamara, Crossley, & McCarthy, 2010) and texts written by L2 writers from diverse proficiency levels (Crossley & McNamara, 2012;Crossley, Salsbury, & McNamara, 2011;Crossley, Salsbury, McNamara, & Jarvis, 2010). The L2 studies showed that lexical diversity and word frequency, both measures of vocabulary size, and word hypernymy (a measure of breadth) had strong relationships with ratings of lexical proficiency, while lexical diversity, word familiarity, frequency, meaningfulness, and imagability (i.e., ease of constructing a mental image) had strong relationships with ratings of text quality.…”

“…Furthermore, lexical features related to breadth, depth, and accessibility also differentiate among L2 texts from writers of different proficiency levels (Crossley & McNamara, 2009, 2012Crossley, Salsbury, et al, 2011). However, these studies did not analyze integrated-writing tasks, but instead used independent writing tasks such as free writing and narratives.…”

A comparative investigation into effects of L1 lexicalization and cultural loadedness on incidental vocabulary acquisition and retention

“…In this regard, vocabulary knowledge (or lexical competence) is widely recognized by many vocabulary acquisition researchers as a fundamental component of language acquisition and communicative competence (Hairrell, Rupley, & Simmons, 2011;Nation, 2013;Ang, 2014). Knowledge of vocabulary is also closely associated with the four traditional languages skills of reading, writing, listening and speaking (Brown, Waring, & Donkaewbua, 2008;Crossley, Salsbury, & McNamara, 2011;Mehrpour & Rahimi, 2010;Karami & Salahshoor, 2014). Therefore, in the realm of second language acquisition (SLA), research on the acquisition and retention of various aspects of vocabulary knowledge are worthwhile lines of investigation (Hairrell et al, 2011;Nation, 2013;Webb, 2007).…”

Predictive modeling of human placement decisions in an English Writing Placement Test

“…Linear prediction in multiple linear regressions (Crossley & McNamara, 2012;Crossley, Kyle, Allen, Guo, & McNamara, 2014;Ferris, 1994;Guo, Crossley, & McNamara, 2013;McNamara, Crossley, & Roscoe, 2013) assumes that a straight line would represent the relationships between written discourse features (i.e., exploratory variables or predictors) and writing proficiency levels (i.e., the response variable). The Linear Discriminant Analysis (LDA) classifier employed in Crossley, Salsbury, and McNamara (2011) attempted to find straight lines that best divide regions of classes among observations (James, Witten, Hastie, & Tibshirani, 2013). However, linear boundaries would not be often the case in reality.…”