The influence of lexical features on teacher judgements of ESL argumentative essays

Vögelin, Cristina; Jansen, Thorben; Keller, Stefan; Machts, Nils; Möller, Jens

doi:10.1016/j.asw.2018.12.003

Cited by 45 publications

(35 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…We do not find evidence that teachers are unduly influenced by differences in language complexity when assigning content grades. This is an encouraging finding in light of Vögelin et al (2019)'s study on the effect of differences in lexical complexity on construct-unrelated grades. Our study differs in several aspects from their set-up: We asked experienced teachers rather than pre-service teachers, and we used the set-up of the Abitur they are familiar with.…”

Section: Discussionsupporting

confidence: 56%

“…There is evidence that mechanical accuracy over-proportionally influences grades and even affects the evaluation of unrelated concepts such as content (Cumming et al, 2002;Rezaei and Lovorn, 2010). Differences in lexical sophistication and diversity have been shown to impact teachers' evaluation of grammar and essay structure (Vögelin et al, 2019). This is a potentially severe issue for the German education system.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Computationally Modeling the Impact of Task-Appropriate Language Complexity and Accuracy on Human Grading of German Essays

Weiß

Riemenschneider

Schröter

et al. 2019

Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications

View full text Add to dashboard Cite

Computational linguistic research on the language complexity of student writing typically involves human ratings as a gold standard. However, educational science shows that teachers find it difficult to identify and cleanly separate accuracy, different aspects of complexity, contents, and structure. In this paper, we therefore explore the use of computational linguistic methods to investigate how taskappropriate complexity and accuracy relate to the grading of overall performance, content performance, and language performance as assigned by teachers. Based on texts written by students for the official school-leaving state examination (Abitur), we show that teachers successfully assign higher language performance grades to essays with higher task-appropriate language complexity and properly separate this from content scores. Yet, accuracy impacts teacher assessment for all grading rubrics, also the content score, overemphasizing the role of accuracy. Our analysis is based on broad computational linguistic modeling of German language complexity and an innovative theory-and datadriven feature aggregation method inferring task-appropriate language complexity.

show abstract

Section: Discussionsupporting

confidence: 56%

Section: Introductionmentioning

confidence: 99%

Computationally Modeling the Impact of Task-Appropriate Language Complexity and Accuracy on Human Grading of German Essays

Weiß

Riemenschneider

Schröter

et al. 2019

Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications

View full text Add to dashboard Cite

show abstract

“…Teachers failed to distinguish between formal errors and the independent quality of content in a student essay. Similarly, Vögelin et al (2018Vögelin et al ( , 2019 found that lexical features and spelling influenced not only holistic teacher judgments of students' writing in English as a second or foreign language, but also their assessment of other analytical criteria (e.g., grammar). Even though these studies do not consider text length as a potential source of bias, they do show that constructirrelevant aspects influence judgments of teachers.…”

Section: Theoretical Backgroundmentioning

confidence: 98%

Is a Long Essay Always a Good Essay? The Effect of Text Length on Writing Assessment

et al. 2020

Self Cite

View full text Add to dashboard Cite

The assessment of text quality is a transdisciplinary issue concerning the research areas of educational assessment, language technology, and classroom instruction. Text length has been found to strongly influence human judgment of text quality. The question of whether text length is a construct-relevant aspect of writing competence or a source of judgment bias has been discussed controversially. This paper used both a correlational and an experimental approach to investigate this question. Secondary analyses were performed on a large-scale dataset with highly trained raters, showing an effect of text length beyond language proficiency. Furthermore, an experimental study found that pre-service teachers tended to undervalue text length when compared to professional ratings. The findings are discussed with respect to the role of training and context in writing assessment.

show abstract

“…Факторы, ориентированные на основание оценки, определяют степень надежности оценочных суждений учителя в зависимости от того, опирается он или нет на критерии оценки [33,40], а также какие качественные параметры продукта учебной деятельности учитывает учитель при его оценке в форме оценочного суждения [33].…”

Section: результаты исследованияunclassified

The problem of the reliability of the teachers’ assessment judgments in formative assessment

Bodony

2020

Vestn. Mininskogo univ.

View full text Add to dashboard Cite

Introduction. The organization of the educational process involves the implementation of ongoing assessment, which provides support for students’ educational activities and the impact on the educational process to improve its effectiveness. One of the elements of the teacher's assessment activity is an assessment judgment, which is formulated by the teacher in the current process of monitoring, diagnosing, analyzing and interpreting the results of students' learning activities and is aimed at informing students about the degree of the planned result achievement, students’ motivation, approval, remark, etc. The study of the teacher's assessment judgments in the context of formative assessment correlates with the problem of reliability, which ensures the consistency, objectivity, fairness and stability of the results of assessment activities.Materials and Methods. The material for the research includes 20 articles published from 2017 to 2020 and devoted to the problems of teachers’ assessment judgments during the process of formative assessment. They were collected by using keywords in the full-text databases of published scientific research which were ScienceDirect and Wiley Online Library. The selected articles were analyzed and summarized from the point of view of considering the factors that influence the teacher's assessment judgments and create conditions for reducing reliability.Results. As a result of the analysis of publications, we identified the following groups of factors that affect the reliability of the teacher's assessment judgments: factors caused by the components of the assessment situation, which include factors related to the subject and object of the assessment, assessment target, scope of the assessment, the basis of the assessment, as well as the use of the assessment tool; factors related to the assessment judgment; external factors.Discussion and Conclusions. The study of the factors influencing the reliability of the teacher's assessment judgments determines the possibilities of increasing the effectiveness of the assessment activities in the process of formative assessment. Understanding the role of each factor creates conditions for preparing recommendations for the teacher to improve the reliability of the current assessment. In addition, on the basis of the classification under consideration, it is possible to conduct empirical studies to clarify the specifics of activating certain factors depending on the goals, conditions and individual characteristics of all participants of the educational process in order to increase the effectiveness of monitoring the achievements of planned educational results and support the educational process.

show abstract

The influence of lexical features on teacher judgements of ESL argumentative essays

Cited by 45 publications

References 54 publications

Computationally Modeling the Impact of Task-Appropriate Language Complexity and Accuracy on Human Grading of German Essays

Computationally Modeling the Impact of Task-Appropriate Language Complexity and Accuracy on Human Grading of German Essays

Is a Long Essay Always a Good Essay? The Effect of Text Length on Writing Assessment

The problem of the reliability of the teachers’ assessment judgments in formative assessment

Contact Info

Product

Resources

About