Countering the Effects of Lead Bias in News Summarization via Multi-Stage Training and Auxiliary Losses

Abstract

Sentence position is a strong feature for news summarization, since the lead often (but not always) summarizes the key points of the article. In this paper, we show that recent neural systems excessively exploit this trend, which although powerful for many inputs, is also detrimental when summarizing documents where important content should be extracted from later parts of the article. We propose two techniques to make systems sensitive to the importance of content in different parts of the article. The first technique employs 'unbiased' data; i.e., randomly shuffled sentences of the source document, to pretrain the model. The second technique uses an auxiliary ROUGEbased loss that encourages the model to distribute importance scores throughout a document by mimicking sentence-level ROUGE scores on the training data. We show that these techniques significantly improve the performance of a competitive reinforcement learning based extractive system, with the auxiliary loss being more powerful than pretraining.

The feasibility of convalescent plasma therapy in severe COVID- 19 patients: a pilot study

“…Furthermore, abstracts are almost always available rather than behind paywalls like full text articles. For news summarisation, we used a state-of-the-art extractive model (Grenander et al, 2019) to extract sentences forming a summary of the original text. This model provides a summary de-biasing mechanism preventing it from focusing on specific parts of the full article, preserving the summary's informational authenticity as much as possible.…”

Multi-XScience: A Large-scale Dataset for Extreme Multi-document Summarization of Scientific Articles

“…The high level of abstractiveness makes our dataset challenging since models cannot simply copy sentences from the reference articles. (Grenander et al, 2019). The extractive oracle performance indicates the level of "extractiveness" of each dataset.…”

“…High positional and extractive biases can undesirably enable models to achieve high summarization scores by copying sentences from certain (fixed) positions, e.g. lead sentences in news summarization (Grenander et al, 2019;Narayan et al, 2018a). Empirical results show that our dataset is challenging and requires models having high-level of text abstractiveness.…”

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop

“…It may be tempting to apply neural abstractive summarization to meetings given its remarkable recent success on summarization benchmarks, e.g., CNN/DM (See et al, 2017;Chen and Bansal, 2018;Gehrmann et al, 2018;Laban et al, 2020). However, the challenge lies not only in handling hallucinations that are seen in abstractive models (Kryscinski et al, 2019;Lebanoff et al, 2019;Maynez et al, 2020) but also the models' strong positional bias that occurs as a consequence of fine-tuning on news articles (Kedzie et al, 2018;Grenander et al, 2019). Neural summarizers also assume a maximum sequence length, e.g., Perez-Beltrachini et al (2019) use the first 800 tokens of the document as input.…”

“…The pretraining data contain 160G of news, books, stories, and web text. It remains unclear if the model can effectively identify salient content on spoken text and, how it is to reduce lead bias that is not as frequent in spoken text as in news writing (Grenander et al, 2019). Secondly, a transcript can far exceed the maximum input length of the model, which is restricted by the GPU memory size.…”