Options for Written Language Assessment

Moran, Mary Ross

doi:10.17161/fec.v19i5.7494

Cited by 5 publications

(2 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Well-known tests such as TOEFL, GRE, ACT, LinguaSkill, Duolingo English Test, and SAT are some examples of CR based standardized testing, where the tests present "naturalistic" prompts such as writing essays and summarizing a news report (such as in GRE, TOEFL, SAT), and making a conversation and giving a speech (such as in Duolingo English Test, TOEFL, LinguaSkill) as opposed to artificial tasks like solving multiple choice questions or filling blanks. The community believes that these types of prompts are much more likely to be encountered by the candidate in real-life scenarios (Stiggins, 1982;Charney, 1984;Messick, 1996) and that they provide a more accurate way of measuring test construct (Moran, 1987;Wiggins, 1991). Moreover, artificial testing like those of multiple choice questions lead to washback effects (refers to the extent to which the introduction and use of a test influences language teachers and learners to do things they would not otherwise do that promote or inhibit language learning) (Messick, 1996;Wiggins, 1991).…”

Section: Automatic Essay Scoringmentioning

confidence: 99%

Automatic Essay Scoring Systems Are Both Overstable And Oversensitive: Explaining Why And Proposing Defenses

Kumar

Parekh

Singh

et al. 2023

dad

View full text Add to dashboard Cite

Deep-learning based Automatic Essay Scoring (AES) systems are being actively used in various high-stake applications in education and testing. However, little research has been put to understand and interpret the black-box nature of deep-learning-based scoring algorithms. While previous studies indicate that scoring models can be easily fooled, in this paper, we explore the reason behind their surprising adversarial brittleness. We utilize recent advances in interpretability to find the extent to which features such as coherence, content, vocabulary, and relevance are important for automated scoring mechanisms. We use this to investigate the oversensitivity (i.e., large change in output score with a little change in input essay content) and overstability (i.e., little change in output scores with large changes in input essay content) of AES. Our results indicate that autoscoring models, despite getting trained as “end-to-end” models with rich contextual embeddings such as BERT, behave like bag-of-words models. A few words determine the essay score without the requirement of any context making the model largely overstable. This is in stark contrast to recent probing studies on pre-trained representation learning models, which show that rich linguistic features such as parts-of-speech and morphology are encoded by them. Further, we also find that the models have learnt dataset biases, making them oversensitive. The presence of a few words with high co-occurrence with a certain score class makes the model associate the essay sample with that score. This causes score changes in ∼95% of samples with an addition of only a few words. To deal with these issues, we propose detection-based protection models that can detect oversensitivity and samples causing overstability with high accuracies. We find that our proposed models are able to detect unusual attribution patterns and flag adversarial samples successfully.

show abstract

Section: Automatic Essay Scoringmentioning

confidence: 99%

Automatic Essay Scoring Systems Are Both Overstable And Oversensitive: Explaining Why And Proposing Defenses

Kumar

Parekh

Singh

et al. 2023

dad

View full text Add to dashboard Cite

show abstract

“…In this regard, a variety of visual or verbal prompts have been used to help individuals begin their writing (e.g., a story starter). Good prompts identify the form of the writing to be generated, describe the specific audience to be addressed, and inform the student of the purpose of the communication (Moran, 1988). However, there have been few data to support the specific kind of prompt that might yield the best results.…”

Section: Assessment Issuesmentioning

confidence: 99%

The Language of Written Language

Hooper¹

2002

J Learn Disabil

View full text Add to dashboard Cite

This special issue includes five articles dealing with selected aspects of writing disorders, crafted by key contributors to the literature on written language. Articles addressing definitional issues and diagnostic questions, assessment strategies, selected neurodevelopmental components of written expression, the manifestation of writing problems in children and young adults, direct intervention, training for elementary teachers in the early grades in the area of literacy, and brain-behavior linkages are presented. This introduction provides an overview of these articles tied to specific issues confronting the field of written language. Some thoughts and directions for the future are articulated.

show abstract

Assessing the Writing Product: Qualitative and Quantitative Measures

Isaacson¹

1988

Exceptional Children

View full text Add to dashboard Cite

Assessment of writing skills must focus on the writing product in order to satisfy four educational functions: identifying unsuccessful writers, planning instruction, monitoring student progress, and giving feedback. Within the last 25 years, various researchers have tested the validity of simple direct measures that correspond to five important aspects of the writing product: fluency, content, conventions, syntax, and vocabulary. Measures for each of these components, both qualitative and quantitative, are discussed.

show abstract

Options for Written Language Assessment

Cited by 5 publications

References 10 publications

Automatic Essay Scoring Systems Are Both Overstable And Oversensitive: Explaining Why And Proposing Defenses

Automatic Essay Scoring Systems Are Both Overstable And Oversensitive: Explaining Why And Proposing Defenses

The Language of Written Language

Assessing the Writing Product: Qualitative and Quantitative Measures

Contact Info

Product

Resources

About