Evaluating the Factual Consistency of Abstractive Text Summarization

Abstract

The most common metrics for assessing summarization algorithms do not account for whether summaries are factually consistent with source documents. We propose a weakly-supervised, model-based approach for verifying factual consistency and identifying conflicts between source documents and generated summaries. Training data is generated by applying a series of rule-based transformations to the sentences of source documents. The factual consistency model is then trained jointly for three tasks: 1) predict whether each summary sentence is factually consistent or not, 2) in either case, extract a span in the source document to support this consistency prediction, 3) for each summary sentence that is deemed inconsistent, extract the inconsistent span from it. Transferring this model to summaries generated by several neural models reveals that this highly scalable approach outperforms previous models, including those trained with strong supervision using datasets from related domains, such as natural language inference and fact checking. Additionally, human evaluation shows that the auxiliary span extraction tasks provide useful assistance in the process of verifying factual consistency. We also release a manually annotated dataset for factual consistency verification, code for training data generation, and trained model weights at https://github.com/salesforce/factCC.

Templated Text Synthesis for Expert-Guided Multi-Label Extraction from Radiology Reports

“…Various approaches have been proposed for text data augmentation, targeting improved performance on some diverse natural language processing (NLP) applications. Synthetic data can be generated using very simple rule-based transformations including noise injection (inserting random words), random word deletion or number swapping [21,22]. Another approach to creating synthetic text data are to randomly split training documents or sentences into multiple training fragments.…”

Reducing Quantity Hallucinations in Abstractive Summarization

“…Another line of research focused on evaluating factual consistency of summarization systems. Kryscinski et al (2019b) proposed a weaklysupervised, model-based approach for evaluating factual consistency between source documents and generated summaries. They first generate training data by applying a series of transformations to randomly selected individual sentences from source documents (which they call claims) and assign them a binary label based on the type of the transformation.…”

Truth or Error? Towards systematic analysis of factual errors in abstractive summaries

“…Similarly, the authors of Falke et al (2019) evaluate three different state-of-the-art systems and find that between 8 and 26% of the generated summaries contain at least one factual error, even though ROUGE scores indicate good performance. Kryściński et al (2019) propose a weakly supervised method for verifying factual consistency between document and summary by training a binary model that predicts whether or not a sentence is consistent. For this purpose they artificially generate a dataset with various types of errors, such as entity or number swapping, paraphrasing, pronoun swapping, sentence negation and noise injection.…”