Automated Pyramid Summarization Evaluation

Gao, Yanjun; Sun, Chen; Passonneau, Rebecca J.

doi:10.18653/v1/k19-1038

Cited by 24 publications

(10 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The automated wise crowd method performs fairly well on these summaries, as described in [64], with a Pearson correlation of 0.66 on the Autonomous Vehicle summaries when comparing the manual and automated summary content assessment, and a Pearson correlation of 0.72 for the Cryptocurrency summaries. Previously, the instructor found the content scores and justifications to be very useful, including cases where the tool gave low scores to written work that, on reflection, were scored favorably based on the writing fluency rather than the content [67].…”

Section: B Content Annotation and Analysis: The Wise Crowd Methodsmentioning

confidence: 79%

“…Originally this method was applied through manual annotation procedure (see next paragraph). An automated approach to the assessment and feedback step has been developed [62] and, more recently, a fully automated approach called PyrEval that identifies and ranks the SCUs from a set of reference summaries, then uses the weighted SCUs to assess new summaries [64], was developed. PyrEval was tested on summaries from the Autonomous Vehicles and Cryptocurrency topics.…”

Section: B Content Annotation and Analysis: The Wise Crowd Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Analytical Techniques for Developing Argumentative Writing in STEM

Davies¹,

Gao²,

Muresan³

et al. 2021

Preprint

View full text Add to dashboard Cite

<p>This article demonstrates how experiential learning could be used to develop argumentative essay writing skills in STEM students. Written feedback, when delivered in a timely manner, is an effective way of advancing students’ understanding of the writing process. Unfortunately, large class sizes and the limited backgrounds of instructors do not always make formative feedback possible. STEM students are especially disadvantaged since approaches to teaching written communication tend to differ from the trial-and-error strategies compatible with many STEM areas. An experiential learning approach to writing instruction can have a positive impact on developing writing skills in STEM learners. Implementing algorithms for providing STEM students with immediate, dependable, formative feedback is expected to improve their performance in writing. This paper discusses an experiential learning project for teaching argumentative writing was delivered to computer science and engineering freshmen. Also discussed are automated analysis of content and argumentation in the essays, using NLP methods.</p>

show abstract

Section: B Content Annotation and Analysis: The Wise Crowd Methodsmentioning

confidence: 79%

Section: B Content Annotation and Analysis: The Wise Crowd Methodsmentioning

confidence: 99%

Analytical Techniques for Developing Argumentative Writing in STEM

Davies¹,

Gao²,

Muresan³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Originally, this method was applied through manual annotation procedure (see next paragraph). An automated approach to the assessment and feedback step was developed [57] and, more recently, a fully automated approach called PyrEval that identifies and ranks the SCUs from a set of reference summaries, then uses the weighted SCUs to assess new summaries [60]. PyrEval was tested on summaries from the AV and Cryptocurrency topics.…”

Section: B Technology For Supporting Experiential Learning 1) Content Annotation and Analysis (The Wise Crowd Method)mentioning

confidence: 99%

“…In this step, which uses SEAView, elementary discourse units (EDUs) are annotated [61], [62]; these are effectively individual clauses or propositions stored in the sep file. In contrast to a summary of a source text, the quality of a student's argument is not expected to depend on how much of The automated wise crowd method performs fairly well on these summaries, as described in [60], with a Pearson correlation of 0.66 on the autonomous vehicle summaries when comparing the manual and automated summary content assessment, and a Pearson correlation of 0.72 for the cryptocurrency summaries. In a previous study [63], the instructor found the content scores and justifications to be very useful in understanding cases where the tool gave lower scores to written work compared to those given by the tutors.…”

Section: B Technology For Supporting Experiential Learning 1) Content Annotation and Analysis (The Wise Crowd Method)mentioning

confidence: 99%

Analytical Techniques for Developing Argumentative Writing in STEM: A Pilot Study

et al. 2022

Self Cite

View full text Add to dashboard Cite

Contribution: Demonstrates how to use experiential learning (EL) to improve argumentative writing. Presents the design and development of a natural language processing (NLP) application for aiding instructors in providing feedback on student essays. Discusses how EL combined with automated support provides an analytical approach to improving written-communication skills.Background: High-quality, timely, feedback is an effective way to improve students' writing. However, large class sizes and limited instructor backgrounds often make formative feedback impossible. Recent trends, including lowering entry requirements, have added to these challenges. Assistive technologies for implementing inclusive education provide viable solutions.Research Questions: 1) How and why can EL be used to develop argumentative writing skills in university STEM students? 2) How can technologies be developed to support using EL in teaching writing? and 3) How might the holistic impact of using such analytic techniques be evaluated?Methodology: Participants in an EL project were assigned two essays in sequence. They were given instructions on making good arguments and shown how to use an analytic rubric to maximize their scores. The essays were hand scored by tutors who provided scores for each dimension of the rubric. Subsequently, the content and argumentation of the essays were analyzed using NLP techniques to obtain independent scores. Qualitative data were also collected.Findings: The project produced transformative writing experiences for the participants. It showed how analytical techniques help improve writing skills and how relevant automated instructor assistance can be developed using NLP technologies.

show abstract

“…Recently, systems have been developed to ease the construction of Pyramid scores, e.g. (Hirao et al, 2018;Yang et al, 2016;Gao et al, 2019b;, but they still require human-annotated Summary Content Units (SCUs) to produce reliable scores. Besides SCUs, recent work has explored eliciting preferences over summaries (Zopf, 2018;Gao et al, 2018Gao et al, , 2019a and annotations of important bi-grams (P.V.S and Meyer, 2017) to derive summary ratings.…”

Section: Related Workmentioning

confidence: 99%

SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization

Gao¹,

Zhao²,

Eger³

2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

We study unsupervised multi-document summarization evaluation metrics, which require neither human-written reference summaries nor human annotations (e.g. preferences, ratings, etc.). We propose SUPERT, which rates the quality of a summary by measuring its semantic similarity with a pseudo reference summary, i.e. selected salient sentences from the source documents, using contextualized embeddings and soft token alignment techniques. Compared to the state-of-theart unsupervised evaluation metrics, SUPERT correlates better with human ratings by 18-39%. Furthermore, we use SUPERT as rewards to guide a neural-based reinforcement learning summarizer, yielding favorable performance compared to the state-of-the-art unsupervised summarizers. All source code is available at https://github.com/yg211/ acl20-ref-free-eval.

show abstract

Automated Pyramid Summarization Evaluation

Cited by 24 publications

References 34 publications

Analytical Techniques for Developing Argumentative Writing in STEM

Analytical Techniques for Developing Argumentative Writing in STEM

Analytical Techniques for Developing Argumentative Writing in STEM: A Pilot Study

SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization

Contact Info

Product

Resources

About