Rubric Reliability and Annotation of Content and Argument in Source-Based Argument Essays

Gao, Yanjun; Driban, Alex; McManus, Brennan; Musi, Elena; Davies, Patricia; Muresan, Smaranda; Passonneau, Rebecca J.

doi:10.18653/v1/w19-4452

Cited by 6 publications

(4 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It could therefore be used by instructors to provide feedback on students' understanding of sources. The same features that have proved useful for automated analysis of argument in previous work [76] are shown to be the most predictive of the feature sets used here as well. In the context of freshman writing courses, especially for STEM students, Work on integrating automated assessment of argumentation and subject matter content is already in progress.…”

Section: Discussionsupporting

confidence: 53%

Analytical Techniques for Developing Argumentative Writing in STEM

Davies¹,

Gao²,

Muresan³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

<p>This article demonstrates how experiential learning could be used to develop argumentative essay writing skills in STEM students. Written feedback, when delivered in a timely manner, is an effective way of advancing students’ understanding of the writing process. Unfortunately, large class sizes and the limited backgrounds of instructors do not always make formative feedback possible. STEM students are especially disadvantaged since approaches to teaching written communication tend to differ from the trial-and-error strategies compatible with many STEM areas. An experiential learning approach to writing instruction can have a positive impact on developing writing skills in STEM learners. Implementing algorithms for providing STEM students with immediate, dependable, formative feedback is expected to improve their performance in writing. This paper discusses an experiential learning project for teaching argumentative writing was delivered to computer science and engineering freshmen. Also discussed are automated analysis of content and argumentation in the essays, using NLP methods.</p>

show abstract

Section: Discussionsupporting

confidence: 53%

Analytical Techniques for Developing Argumentative Writing in STEM

Davies¹,

Gao²,

Muresan³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The results of the automated analysis of the student summaries show that the automated summary analysis performs well and could, therefore, be used by instructors to provide feedback on students' understanding of sources. The same features that have proved useful for automated analysis of argument in previous work [73] are shown to be the most predictive of the feature sets used here, in the context of undergraduate writing courses designed for STEM students. The work on integrating automated assessment of argumentation and subject matter content is already in progress.…”

Section: Discussionmentioning

confidence: 58%

Analytical Techniques for Developing Argumentative Writing in STEM: A Pilot Study

et al. 2022

Self Cite

View full text Add to dashboard Cite

Contribution: Demonstrates how to use experiential learning (EL) to improve argumentative writing. Presents the design and development of a natural language processing (NLP) application for aiding instructors in providing feedback on student essays. Discusses how EL combined with automated support provides an analytical approach to improving written-communication skills.Background: High-quality, timely, feedback is an effective way to improve students' writing. However, large class sizes and limited instructor backgrounds often make formative feedback impossible. Recent trends, including lowering entry requirements, have added to these challenges. Assistive technologies for implementing inclusive education provide viable solutions.Research Questions: 1) How and why can EL be used to develop argumentative writing skills in university STEM students? 2) How can technologies be developed to support using EL in teaching writing? and 3) How might the holistic impact of using such analytic techniques be evaluated?Methodology: Participants in an EL project were assigned two essays in sequence. They were given instructions on making good arguments and shown how to use an analytic rubric to maximize their scores. The essays were hand scored by tutors who provided scores for each dimension of the rubric. Subsequently, the content and argumentation of the essays were analyzed using NLP techniques to obtain independent scores. Qualitative data were also collected.Findings: The project produced transformative writing experiences for the participants. It showed how analytical techniques help improve writing skills and how relevant automated instructor assistance can be developed using NLP technologies.

show abstract

“…In content-driven programs where subject specialists teach their subject matter, students’ compositions are predominantly evaluated on their content. Rubrics employed in these courses commonly entail multiple specific content-related criteria, without dividing linguistic features into minute elements as frequently seen in L2 writing assessment and language-driven CLIL (e.g., Bean & Melzer, 2021; Bukhari et al, 2021; Gao et al, 2019; Garza et al, 2021; Walvoord & Anderson, 2010). For example, the following nine criteria exist in a rubric used at two U.S. universities to assess first-year students’ compositions, including engagement with external sources, originality of the contention, clarity of the contention, effectiveness of supporting evidence, presentation of the writer’s own idea, organization, source use, language style, use of standard written English, and formatting (Walvoord & Anderson, 2010).…”

Section: Literature Reviewmentioning

confidence: 99%

Assessing the content quality of essays in content and language integrated learning: Exploring the construct from subject specialists’ perspectives

Sato

2023

Language Testing

View full text Add to dashboard Cite

Assessing the content of learners’ compositions is a common practice in second language (L2) writing assessment. However, the construct definition of content in L2 writing assessment potentially underrepresents the target competence in content and language integrated learning (CLIL), which aims to foster not only L2 proficiency but also critical thinking skills and subject knowledge. This study aims to conceptualize the construct of content in CLIL by exploring subject specialists’ perspectives on essays’ content quality in a CLIL context. Eleven researchers of English as a lingua franca (ELF) rated the content quality of research-based argumentative essays on ELF submitted in a CLIL course and produced think-aloud protocols. This study explored some essay features that have not been considered relevant in language assessment but are essential in the CLIL context, including the accuracy of the content, presence and quality of research, and presence of elements required in academic essays. Furthermore, the findings of this study confirmed that the components of content often addressed in language assessment (e.g., elaboration and logicality) are pertinent to writing assessment in CLIL. The manner in which subject specialists construe the content quality of essays on their specialized discipline can deepen the current understanding of content in CLIL.

show abstract

Rubric Reliability and Annotation of Content and Argument in Source-Based Argument Essays

Cited by 6 publications

References 31 publications

Analytical Techniques for Developing Argumentative Writing in STEM

Analytical Techniques for Developing Argumentative Writing in STEM

Analytical Techniques for Developing Argumentative Writing in STEM: A Pilot Study

Assessing the content quality of essays in content and language integrated learning: Exploring the construct from subject specialists’ perspectives

Contact Info

Product

Resources

About