A Supervised Learning Model for the Automatic Assessment of Language Levels Based on Learner Errors

Ballier, Nicolas; Gaillat, Thomas; Simpkin, Andrew; Stearns, Bernardo; Bouyé, Manon; Zarrouk, Manel

doi:10.1007/978-3-030-29736-7_23

Cited by 8 publications

(6 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We envisage various strategies for automatically assessing ESL learners' written proficiency. Firstly, the use of error-rate as a feature may give rather significant results both if used as the only feature, as illustrated by Ballier et al (2019), or if used in combination with other linguistic features, as shown in Yannakoudakis et al (2011), but unlike such studies -in which shallow learning algorithms were employed -we propose to investigate the application of deep neural networks. Secondly, the use of two input texts -one containing answers with errors, the other one containing the corrected versions of the answers -may also result in an efficient approach.…”

Section: Discussionmentioning

confidence: 99%

“…This work was reproduced by Caines and Buttery (2020) who applied such experiments also to English and Spanish corpora. Recently, the work described by Ballier et al (2019) has investigated the possibility of predicting CEFR proficiency levels based on manually annotated errors in the EF-Cambridge Open Language Database (EFCAMDAT) corpus. Their classifier based on a random forest model achieved 70% accuracy.…”

Section: State Of the Artmentioning

confidence: 99%

See 1 more Smart Citation

Survey datasets of VIKKE assessment tool on children's Finnish Sign Language development

Syrj�l�,

Kanto

2024

Jyx

View full text Add to dashboard Cite

Taking as a case in point Cypriot Arabic (CyAr), a severely endangered language spoken in Cyprus, this paper discusses the results of the research project MapCyArS (Mapping Cypriot Arabic speakers: An investigation into linguistic demography and the sociolinguistic profile of Kormakiote Maronites), funded by the A G Leventis Foundation at the University of Cyprus (2017-2020), and highlights the need to change the way we consider language testing and assessment to fit the sociolinguistic situation of less widely spoken, oral and/or endangered languages and their communities. In the case of CyAr, we need to consider the fact that it is a language of oral tradition that has suffered severe reduction on all linguistic levels and it is not a first language for people under 40 anymore.1 Although the CEFR is designed also for lesser used and taught languages, it has nevertheless been applied to very few, among them: Catalan, Galician, Basque, Friulan or Welsh, etc., which are officially recognized within their respective areas of use. Conversation Verbal L2 vs Verbal L1The task consists of using CyGr or translanguaging to provide short answers to questions asked in CyAr LEVEL 1 LEVEL 0 Fails to understand CyAr and correctly respond in SG 3 /(Cy)Gr Correctly understands CyAr and responds in SG/(Cy)Gr 5 Conversation Verbal L2 vs Verbal L2 The task consists of using CyAr to provide short answers to questions asked in CyAr LEVEL 1 LEVEL 2 LEVEL 0 Fails to understand and correctly respond in CyAr Correctly understands common sentences and responds in CyAr Correctly understands a wide range of conversation 6 Metalinguistic/ Grammatical Verbal L2 vs Verbal L2The task consists of completing orally metalinguistic or grammatical tasks such as declining a verb, giving the plural of a noun, translating etc. LEVEL 1 LEVEL 2 LEVEL 0Unable to accomplish any metalinguistic task Able to accomplish simple metalinguistic tasks Able to accomplish complex metalinguistic tasks Conclusion 3Designing an assessment test for an endangered oral language like CyAr is a complex objective that needs to take into consideration the oral character of CyAr, its endangered situation and the advanced state of its replacement by (Cy)Gr.The oral character of CyAr suggests that we move away from reading and writing as means to evaluate oral skills. Its endangered situation suggests that traditional approaches in testing and assessment by level of competence (A1, A2, etc.) should be abandoned in favour of a more flexible type of evaluation. The advanced state of CyAr replacement by (Cy)Gr leads towards a more flexible use of language resources to evaluate speakers' proficiency, and invites us to capitalize on the use of (Cy)Gr or translanguaging to do so. The project is, therefore, also an opportunity to re-examine the baselines of language assessment used for modern languages in order to adapt them to conditions of orality and severe endangerment.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: State Of the Artmentioning

confidence: 99%

Survey datasets of VIKKE assessment tool on children's Finnish Sign Language development

Syrj�l�,

Kanto

2024

Jyx

View full text Add to dashboard Cite

show abstract

“…Recently, the work described by Ballier et al (2019) has investigated the possibility of predicting CEFR proficiency levels based on manually annotated errors in the French and Spanish section of the EFCAMDAT corpus, but their study did not employ deep learning techniques. However, they identified that certain types of errors, such as punctuation, spelling and verb tense errors, are characteristic of specific CEFR proficiency levels.…”

Section: Reference To Prior Workmentioning

confidence: 99%

Cross-corpora experiments of automatic proficiency assessment and error detection for spoken English

Bannò¹,

Matassoni²

2022

Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022)

View full text Add to dashboard Cite

The growing demand for learning English as a second language has led to an increasing interest in automatic approaches for assessing spoken language proficiency. One of the most significant challenges in this field is the lack of publicly available annotated spoken data. Another common issue is the lack of consistency and coherence in human assessment. To tackle both problems, in this paper we address the task of automatically predicting the scores of spoken test responses of Englishas-a-second-language learners by training neural models on written data and using the presence of grammatical errors as a feature, as they can be considered consistent indicators of proficiency through their distribution and frequency. Specifically, we train a feature extractor on EF-CAMDAT, a large written corpus containing error annotations and proficiency levels assigned by human experts, in order to extract information related to grammatical errors and, in turn, we use the resulting model for inference on the CLC-FCE corpus, on the ICNALE corpus, and on the spoken section of the TLT-school corpus, a collection of proficiency tests taken by Italian students. The work investigates the impact of the feature extractor on spoken proficiency assessment as well as the written-to-spoken approach. We find that our error-based approach can be beneficial for assessing spoken proficiency. The results obtained on the considered datasets are discussed and evaluated with appropriate metrics.

show abstract

“…Errors, or negative properties of interlanguage, were also tested. Ballier et al, (2019) showed that error-tag frequencies could be used as potential proficiency predictors.…”

Section: Theoretical Background 21 a Multidimensional Set Of 'Criterial Features'mentioning

confidence: 99%

“…Ballier et al . (2019) showed that error-tag frequencies could be used as potential proficiency predictors.…”

Section: Theoretical Backgroundmentioning

confidence: 99%

Predicting CEFR levels in learners of English: The use of microsystem criterial features in a machine learning approach

Gaillat¹,

Simpkin²,

Ballier³

et al. 2021

ReCALL

Self Cite

View full text Add to dashboard Cite

This paper focuses on automatically assessing language proficiency levels according to linguistic complexity in learner English. We implement a supervised learning approach as part of an automatic essay scoring system. The objective is to uncover Common European Framework of Reference for Languages (CEFR) criterial features in writings by learners of English as a foreign language. Our method relies on the concept of microsystems with features related to learner-specific linguistic systems in which several forms operate paradigmatically. Results on internal data show that different microsystems help classify writings from A1 to C2 levels (82% balanced accuracy). Overall results on external data show that a combination of lexical, syntactic, cohesive and accuracy features yields the most efficient classification across several corpora (59.2% balanced accuracy).

show abstract

A Supervised Learning Model for the Automatic Assessment of Language Levels Based on Learner Errors

Cited by 8 publications

References 20 publications

Survey datasets of VIKKE assessment tool on children's Finnish Sign Language development

Survey datasets of VIKKE assessment tool on children's Finnish Sign Language development

Cross-corpora experiments of automatic proficiency assessment and error detection for spoken English

Predicting CEFR levels in learners of English: The use of microsystem criterial features in a machine learning approach

Contact Info

Product

Resources

About