Demoting the Lead Bias in News Summarization via Alternating Adversarial Learning

Abstract

In news articles the lead bias is a common phenomenon that usually dominates the learning signals for neural extractive summarizers, severely limiting their performance on data with different or even no bias. In this paper, we introduce a novel technique 1 to demote lead bias and make the summarizer focus more on the content semantics. Experiments on two news corpora with different degrees of lead bias show that our method can effectively demote the model's learned lead bias and improve its generality on out-ofdistribution data, with little to no performance loss on in-distribution data.

Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Models

“…Taking a closer look at the generated dependency structures presented on the right side in Figure 3, the alignment between PLM inferred discourse trees and the simple chain structure is predominant, suggesting a potential weakness in regards to the discourse captured in the BERT and BART model. Not surprisingly, the highest overlap between PLM-generated trees and the chain structure occurs when finetuning on the CNN-DM dataset, well-known to contain a strong lead-bias (Xing et al, 2021).…”