ARMAN: Pre-training with Semantically Selecting and Reordering of Sentences for Persian Abstractive Summarization

Abstract

ive text summarization is one of the areas influenced by the emergence of pre-trained language models. Current pre-training works in abstractive summarization give more points to the summaries with more words in common with the main text and pay less attention to the semantic similarity between generated sentences and the original document. We propose ARMAN, a Transformer-based encoderdecoder model pre-trained with three novel objectives to address this issue. In ARMAN, salient sentences from a document are selected according to a modified semantic score to be masked and form a pseudo summary. To summarize more accurately and similar to human writing patterns, we applied modified sentence reordering. We evaluated our proposed models on six downstream Persian summarization tasks. Experimental results show that our proposed model achieves state-of-the-art performance on all six summarization tasks measured by ROUGE and BERTScore. Our models also outperform prior works in textual entailment, question paraphrasing, and multiple choice question answering. Finally, we established a human evaluation and show that using the semantic score significantly improves summarization results.

Enhancing Persian Text Summarization Using the mT5 Transformer Model:A Three-Phased Fine-Tuning Approach and Reinforcement Learning

“…This novel model, aptly named "BERT2BERT", not only establishes a solid foundation for future advancements in Persian text summarization but also opens new horizons for natural language understanding and generation in the Persian language domain. Salemi et al 13 have made a substantial contribution to the field of Persian text summarization through their model known as ARMAN . ARMAN adopts a Transformer-based encoder-decoder architecture, distinguishing itself through innovative pretraining objectives tailored specifically for Persian summarization tasks.…”

“…Additionally, we conduct a comparative analysis to benchmark our model's performance in the task of Persian text summarization against established baseline models. These baseline models include BERT2BERT model 12 and ARMAN model 13 . This comparative approach allows us to gain valuable insights into the advancements achieved by our model and its proficiency in generating context-aware and coherent summaries, setting the stage for a more comprehensive evaluation.…”