BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

Abstract

We present BART, a denoising autoencoder for pretraining sequence-to-sequence models. BART is trained by (1) corrupting text with an arbitrary noising function, and (2) learning a model to reconstruct the original text. It uses a standard Tranformer-based neural machine translation architecture which, despite its simplicity, can be seen as generalizing BERT (due to the bidirectional encoder), GPT (with the left-to-right decoder), and other recent pretraining schemes. We evaluate a number of noising approaches, finding the best performance by both randomly shuffling the order of sentences and using a novel in-filling scheme, where spans of text are replaced with a single mask token. BART is particularly effective when fine tuned for text generation but also works well for comprehension tasks. It matches the performance of RoBERTa on GLUE and SQuAD, and achieves new stateof-the-art results on a range of abstractive dialogue, question answering, and summarization tasks, with gains of up to 3.5 ROUGE. BART also provides a 1.1 BLEU increase over a back-translation system for machine translation, with only target language pretraining. We also replicate other pretraining schemes within the BART framework, to understand their effect on end-task performance. 1

Neural Abstractive Text Summarization with Sequence-to-Sequence Models

“…It has achieved superior performance in machine translation task with significantly less training time. Currently, large Transformers [32,133,149], which are pre-trained on a massive text corpus with self-supervised objectives, have achieved superior results in a variety of downstream NLP tasks such as machine understanding [32,84], question-answering [27,86], and abstractive text summarization [34,72,85,112,148,153]. Zhang et al [153] demonstrated that their pre-trained encoder-decoder model can outperform previous state-of-the-art results [28,36,44,63,67,99,100,119,121] on several datasets by fine-tuning with limited supervised examples, which shows that pre-trained models are promising candidates in zero-shot and low-resource summarization tasks.…”

Generating Fact Checking Briefs

“…We use fairseq-py (Ott et al, 2019) to train the QABRIEFER. We use the open-sourced BART model (Lewis et al, 2019) and suggested finetuning hyperparameters, training for 10 epochs and taking the best epoch by validation loss. To generate, we use beam search with beam size 5.…”

Multilingual AMR-to-Text Generation

“…To improve the quality of the encoder, we incorporate large-scale pretraining on millions of sequences of AMR by adopting the generative pretraining approach proposed in Lewis et al (2019a). This pretraining incorporates various noise operations, such as masking (Devlin et al, 2019), span masking (Fan et al, 2019a), and shuffling.…”