A comparison of automated and manual analyses of syntactic complexity in L2 English writing

Abstract

Automated tools for syntactic complexity measurement are increasingly used for analyzing various kinds of second language corpora, even though these tools were originally developed and tested for texts produced by advanced learners. This study investigates the reliability of automated complexity measurement for beginner and lower-intermediate L2 English data by comparing manual and automated analyses of a corpus of 80 texts written by Dutch-speaking learners. Our quantitative and qualitative analyses reveal that the reliability of automated complexity measurement is substantially affected by learner errors, parser errors, and Tregex pattern undergeneration. We also demonstrate the importance of aligning the definitions of analytical units between the computational tool and human annotators. In order to enhance the reliability of automated analyses, it is recommended that certain modifications are made to the system, and non-advanced L2 English data are preprocessed prior to automated analyses.

Utility of Kolmogorov complexity measures: Analysis of L2 groups and L1 backgrounds

“…This observed variability in learner language can defy clear segmentation into clauses, sentences, or T-units, influencing by this the reliability of some complexity measures particularly when computed manually. A similar influence of learner error on the reliability of automated complexity metrics has been found in a study by Châu and Bulté [ 60 ]. This study compared automated (L2SCA/TAASSC) and manual analyses of syntactic complexity in beginner and lower-intermediate level L2 English texts.…”

“…Common computational tools for complexity metrics in L2 writing research are either tailored to specific languages e.g., [34][35][36][37] or may not reliably segment learner language especially at the lower proficiency levels [59], e.g., [60]. The language specificity problem has facilitated the study of writing complexity in predominant and resource-rich languages such as L2 English to the exclusion of other languages.…”

“…In addition to the language-specificity issue of automatic complexity tools, there is some evidence that traditional units of measures in complexity research (e.g., clauses, sentences, t-units) can be difficult to delineate, especially when dealing with written learner language [ 59 ], e.g., [ 60 ]. A manual analysis of complexity measures in L2 Finnish writings suggested that learners’ written production may deviate from those of native speakers, which makes it difficult to categorize their language into rigid production units [ 59 ].…”