Factual Error Correction for Abstractive Summarization Models

Cao, Meng; Dong, Yue; Wu, Jiapeng; Cheung, Jackie Chi Kit

doi:10.18653/v1/2020.emnlp-main.506

Cited by 88 publications

(98 citation statements)

References 13 publications

(26 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…First, while synthetic data generation approaches are specifically designed for factuality evaluation, do these align with actual errors made by generation models? We find the answer is no: techniques using surface-level data corruption (Kryscinski et al, 2020;Zhao et al, 2020;Cao et al, 2020) or paraphrasing (Goyal and Durrett, 2020a) target inherently different error distributions than those seen in actual model generations, and factuality models trained on these datasets perform poorly in practice. Furthermore, we show that different summarization domains, CNN/Daily Mail (Hermann et al, 2015;Nallapati et al, 2016) and XSum (Narayan et al, 2018) (which differ in the style of summaries and degree of abstraction), exhibit substantially different error distributions in generated summaries, and the same dataset creation approach cannot be used across the board.…”

Section: Introductionmentioning

confidence: 95%

“…A recent thread of work has focused on leveraging synthetic data transformations for evaluating factuality (Kryscinski et al, 2020), imposing decoding-time constraints (Zhao et al, 2020), or post-correction of summaries (Cao et al, 2020). Each of these approaches assumes that corruption strategies will yield useful non-factual summaries, while gold summaries are treated as factual.…”

Section: Entity-centric Synthetic Data (Ent-c)mentioning

confidence: 99%

“…Localizing errors potentially allows for post-hoc correction (Zhao et al, 2020;Cao et al, 2020); however, repairing a summary to be fully factual is a very hard problem and past work has focused on a subset of errors as a result. Instead, we show that even our imperfect error localization techniques can be used to meaningfully improve the training data for summarization.…”

Section: Downstream Applicationsmentioning

confidence: 99%

See 2 more Smart Citations

Annotating and Modeling Fine-grained Factuality in Summarization

Goyal¹,

Durrett²

2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Recent pre-trained abstractive summarization systems have started to achieve credible performance, but a major barrier to their use in practice is their propensity to output summaries that are not faithful to the input and that contain factual errors. While a number of annotated datasets and statistical models for assessing factuality have been explored, there is no clear picture of what errors are most important to target or where current techniques are succeeding and failing. We explore both synthetic and human-labeled data sources for training models to identify factual errors in summarization, and study factuality at the word-, dependency-, and sentence-level. Our observations are threefold. First, exhibited factual errors differ significantly across datasets, and commonly-used training sets of simple synthetic errors do not reflect errors made on abstractive datasets like XSUM. Second, human-labeled data with fine-grained annotations provides a more effective training signal than sentence-level annotations or synthetic data. Finally, we show that our best factuality detection model enables training of more factual XSUM summarization models by allowing us to identify non-factual tokens in the training data. 1 Reference Summary: An early-medieval gold pendant created from an imitation of a Byzantine coin that was found in a Norfolk field is a "rare find", a museum expert has said.Source Article Fragment: Discovered on land at North Elmham, near Dereham, the circa 600 AD coin was created by French rulers of the time to increase their available currency. […] The pendant was declared treasure by the Norfolk coroner on Wednesday.An 18th century coin believed to be worth more than #1m has been discovered.A gold pendant created from a necklace was found in a field Entitycentric (Ent-C)The pendant was declared a treasure by the Norfolk coroner on Wednesday.The pendant was declared a treasure by the Ohio coroner on March. Generationcentric (Gen-C) Label Human Annotation non-factual span non-factual arc (factual arcs not shown) Sentences Training Dataset

show abstract

Section: Introductionmentioning

confidence: 95%

Section: Entity-centric Synthetic Data (Ent-c)mentioning

confidence: 99%

Section: Downstream Applicationsmentioning

confidence: 99%

See 1 more Smart Citation

Annotating and Modeling Fine-grained Factuality in Summarization

Goyal¹,

Durrett²

2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

show abstract

“…To correct factual errors, uses pre-trained NLU models to rectify one or more wrong entities in the summary. Concurrent to our work, Cao et al (2020) employs the generation model BART to produce corrected summaries.…”

Section: Fact-aware Summarizationmentioning

confidence: 99%

Enhancing Factual Consistency of Abstractive Summarization

Zhu¹,

Hinthorn²,

Xu³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Automatic abstractive summaries are found to often distort or fabricate facts in the article. This inconsistency between summary and original text has seriously impacted its applicability. We propose a fact-aware summarization model FASUM to extract and integrate factual relations into the summary generation process via graph attention. We then design a factual corrector model FC to automatically correct factual errors from summaries generated by existing systems. Empirical results 1 show that the fact-aware summarization can produce abstractive summaries with higher factual consistency compared with existing systems, and the correction model improves the factual consistency of given summaries via modifying only a few keywords.

show abstract

“…Modern text generation models are known to hallucinate facts (Huang et al, 2020), which has led the community to create models to detect and correct hallucinations (Cao et al, 2020;.…”

Section: Inaccuracy Guardrailmentioning

confidence: 99%

Keep It Simple: Unsupervised Simplification of Multi-Paragraph Text

Laban¹,

Schnabel²,

Bennett³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

This work presents Keep it Simple (KiS), a new approach to unsupervised text simplification which learns to balance a reward across three properties: fluency, salience and simplicity. We train the model with a novel algorithm to optimize the reward (k-SCST), in which the model proposes several candidate simplifications, computes each candidate's reward, and encourages candidates that outperform the mean reward. Finally, we propose a realistic text comprehension task as an evaluation method for text simplification. When tested on the English news domain, the KiS model outperforms strong supervised baselines by more than 4 SARI points, and can help people complete a comprehension task an average of 18% faster while retaining accuracy, when compared to the original text.

show abstract

Factual Error Correction for Abstractive Summarization Models

Cited by 88 publications

References 13 publications

Annotating and Modeling Fine-grained Factuality in Summarization

Annotating and Modeling Fine-grained Factuality in Summarization

Enhancing Factual Consistency of Abstractive Summarization

Keep It Simple: Unsupervised Simplification of Multi-Paragraph Text

Contact Info

Product

Resources

About