6 Automatically Assessing Lexical Sophistication Using Word, Bigram, and Dependency Indices

Shaping Writing Grades

“…also reported that parser precision averaged 80 per cent and recall averaged 85 per cent for amod dependencies, while for dobj dependencies, precision averaged 76 per cent and recall averaged 24 per cent with texts written by English school children. Similarly, high rates of precision and recall were also found in Kyle and Eguchi (2021), who reported an average accuracy rate for noun-adjective dependencies of 96.9 per cent, verb-adverb dependencies had an accuracy rate of 98.6 per cent, verb-direct object dependencies of 96 per cent, and verb-subject 95.4 per cent. Table 9 also shows that second language scores were consistently lower than first language scores.…”

“…Scholars have relied on a narrow set of measures that have been restricted to association measures used in the language learning/assessment literature with the t-score and MI featuring prominently. There has been sparse mention of alternatives or an awareness of how the hundreds of other association measures touted in the literature align with the MI or t-score or may be able to illuminate different collocation properties to those highlighted by the MI and t-score (e.g., see criticisms in Öksuz et al (2021) and acknowledgement of the hundreds of measures in Pecina (2005Pecina ( , 2010, Wiechmann (2008), Gries and Ellis (2015), and more recently Kyle et al (2018) and Kyle and Eguchi (2021)). This Element's starting position is that the use of these measures needs to be understood against the wider bank of association measures that researchers have access to.…”

Assessing spoken lexical and lexicogrammatical proficiency using features of word, bigram, and dependency bigram use

“…Research on the relationship between proficiency and collocation use has generally indicated that more proficient language users tend to use more strongly associated word combinations than less proficient users. Although some research has focused on the degree to which L2 language users produce particular collocations (Durrant & Schmitt, 2009), much of the recent research has focused on n ‐gram collocations (Bestgen & Granger, 2014; Eguchi & Kyle, 2020; Garner et al., 2019) and syntactic dependency collocations (Kyle & Eguchi, 2021; Paquot, 2018; Rubin et al., 2021).…”

“…This analysis suggested (at least in the context of TOEFL iBT argumentative essays) that bigram SOA indices do not add any explanatory power to models that include word‐level and dependency bigram indices. One potential limitation of Kyle and Eguchi (2021) is that, unlike a number of studies that have used the Tool for the Automatic Analysis of Lexical Sophistication (TAALES), the authors used a normed frequency cutoff for bigram selection at 1 per million in the reference corpus (TAALES indices use a simple cutoff of five occurrences), which may have limited the range of SOA scores, and consequently may have resulted in weaker results than in previous studies.…”

A Dependency Treebank of Spoken Second Language English

“…For more precise insights into linguistic development, some studies have constrained the lexical combinations that are used (e.g., adjective + noun or noun + noun combinations). Even more recently, researchers have begun to use dependency parses to analyze lexical items in particular grammatical relationships (e.g., verb + object; Kyle & Eguchi, 2021;Paquot, 2018Paquot, , 2019Rubin, 2021). Lexicogrammatical features: A number of studies have investigated the relationship between L2 proficiency and the use of lexicogrammatical features that are common in academic writing such as various types of noun phrase elaboration (e.g., Biber et al, 2014;Grant & Ginther, 2000;Picoral et al, 2021).…”