Analyzing Sentence Fusion in Abstractive Summarization

Lebanoff, Logan; Muchovej, John; Dernoncourt, Franck; Kim, Doo Soon; Kim, Seokhwan; Chang, Walter; Liu, Fei

doi:10.18653/v1/d19-5413

Cited by 36 publications

(33 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recall that we saw a relatively low agreement among annotators as to whether a sentence contains an error at all. This is in line with observations made by Lebanoff et al (2019), who noted a relatively low inter-annotator agreement for binary faithfulness annotation. However, more investigation is needed.…”

Section: Future Worksupporting

confidence: 92%

“…In contrast, Lebanoff et al (2019) are interested in what happens when summarization systems fuse sentences from the source. They automatically extract fused summary sentences generated by five different systems and conduct a manual annotation of faithfulness and grammaticality using crowd sourcing.…”

Section: Factual Errors In Summariesmentioning

confidence: 99%

“…We find that previous research has not established a detailed typology of summarization errors. Most work instead relies on on a binary distinction between correct and erroneous (Cao et al, 2018;Falke et al, 2019;Lebanoff et al, 2019) or faithfulness measured on a Likert scale (Goodrich et al, 2019). However, not all errors are created equal.…”

Section: Factual Errors In Summariesmentioning

confidence: 99%

See 2 more Smart Citations

Truth or Error? Towards systematic analysis of factual errors in abstractive summaries

Lux¹,

Sappelli²,

Larson³

2020

Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems

View full text Add to dashboard Cite

This paper presents a typology of errors produced by automatic summarization systems. The typology was created by manually analyzing the output of four recent neural summarization systems. Our work is motivated by the growing awareness of the need for better summary evaluation methods that go beyond conventional overlap-based metrics. Our typology is structured into two dimensions. First, the Mapping Dimension describes surface-level errors and provides insight into word-sequence transformation issues. Second, the Meaning Dimension describes issues related to interpretation and provides insight into breakdowns in truth, i.e., factual faithfulness to the original text. Comparative analysis revealed that two neural summarization systems leveraging pretrained models have an advantage in decreasing grammaticality errors, but not necessarily factual errors. We also discuss the importance of ensuring that summary length and abstractiveness do not interfere with evaluating summary quality.

show abstract

Section: Future Worksupporting

confidence: 92%

Section: Factual Errors In Summariesmentioning

confidence: 99%

Section: Factual Errors In Summariesmentioning

confidence: 99%

See 1 more Smart Citation

Truth or Error? Towards systematic analysis of factual errors in abstractive summaries

Lux¹,

Sappelli²,

Larson³

2020

Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems

View full text Add to dashboard Cite

show abstract

“…A summary is reliable only if it is true-to-original. Abstractive summarizers are considered to be less reliable despite their impressive performance on benchmark datasets, because they can hallucinate facts and struggle to keep the original meanings intact (Kryscinski et al, 2019;Lebanoff et al, 2019). In this paper, we seek to generate summary highlights to be overlaid on the original documents to allow summaries to be understood in context and avoid misdirecting readers to false conclusions.…”

Section: Introductionmentioning

confidence: 99%

Better Highlighting: Creating Sub-Sentence Summary Highlights

Cho

Song

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

Self Cite

View full text Add to dashboard Cite

Amongst the best means to summarize is highlighting. In this paper, we aim to generate summary highlights to be overlaid on the original documents to make it easier for readers to sift through a large amount of text. The method allows summaries to be understood in context to prevent a summarizer from distorting the original meaning, of which abstractive summarizers usually fall short. In particular, we present a new method to produce self-contained highlights that are understandable on their own to avoid confusion. Our method combines determinantal point processes and deep contextualized representations to identify an optimal set of sub-sentence segments that are both important and non-redundant to form summary highlights. To demonstrate the flexibility and modeling power of our method, we conduct extensive experiments on summarization datasets. Our analysis provides evidence that highlighting is a promising avenue of research towards future summarization.

show abstract

“…A summarizer, however, is not rewarded for correctly fusing sentences. In fact, when examined more closely, only few sentences in system abstracts are generated by fusion (Falke et al, 2019;Lebanoff et al, 2019). For instance, 6% of summary sentences generated by Pointer-Gen (See et al, 2017) are through fusion, whereas human abstracts contain 32% fusion sentences.…”

Section: Introductionmentioning

confidence: 99%

Learning to Fuse Sentences with Transformers for Summarization

Lebanoff

Dernoncourt

Kim

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

Self Cite

View full text Add to dashboard Cite

The ability to fuse sentences is highly attractive for summarization systems because it is an essential step to produce succinct abstracts. However, to date, summarizers can fail on fusing sentences. They tend to produce few summary sentences by fusion or generate incorrect fusions that lead the summary to fail to retain the original meaning. In this paper, we explore the ability of Transformers to fuse sentences and propose novel algorithms to enhance their ability to perform sentence fusion by leveraging the knowledge of points of correspondence between sentences. Through extensive experiments, we investigate the effects of different design choices on Transformer's performance. Our findings highlight the importance of modeling points of correspondence between sentences for effective sentence fusion.

show abstract

Analyzing Sentence Fusion in Abstractive Summarization

Cited by 36 publications

References 24 publications

Truth or Error? Towards systematic analysis of factual errors in abstractive summaries

Truth or Error? Towards systematic analysis of factual errors in abstractive summaries

Better Highlighting: Creating Sub-Sentence Summary Highlights

Learning to Fuse Sentences with Transformers for Summarization

Contact Info

Product

Resources

About