BART-IT: An Efficient Sequence-to-Sequence Model for Italian Text Summarization

Abstract

The emergence of attention-based architectures has led to significant improvements in the performance of neural sequence-to-sequence models for text summarization. Although these models have proved to be effective in summarizing English-written documents, their portability to other languages is limited thus leaving plenty of room for improvement. In this paper, we present BART-IT, a sequence-to-sequence model, based on the BART architecture that is specifically tailored to the Italian language. The model is pre-trained on a large corpus of Italian-written pieces of text to learn language-specific features and then fine-tuned on several benchmark datasets established for abstractive summarization. The experimental results show that BART-IT outperforms other state-of-the-art models in terms of ROUGE scores in spite of a significantly smaller number of parameters. The use of BART-IT can foster the development of interesting NLP applications for the Italian language. Beyond releasing the model to the research community to foster further research and applications, we also discuss the ethical implications behind the use of abstractive summarization models.

Enhancing Text Summarization with a T5 Model and Bayesian Optimization

“…In summary, the text has a significant development starting from the use of important features such as frequency, word count, and word similarity [19,33]. The advantages of the abstractive approach can remove words that are considered unimportant, while the extractive approach performs summaries based on different phrases in the input data [34]. In the Transformer-based Text-to-Text Transfer Transformer or T5 model, a lot of text summaries are carried out with an abstract approach such as that done by Patwardhan et al [35], Cheng and Yu [36], and Mars [37] which goes through several stages such as tokenization, formation of input-output data, Pretraining and Fine-Tuning, Encoder-Decoder Transformation, and text generation.…”

“…deep transformers that perform the hyperparameter combinations of the T5 model. Another study by La Quatra and Cagliero [34] used the BERT model to produce text summaries. In conducting text summaries, there are many challenges faced, one of which is the structure of text data, currently the resulting text data is unstructured and semi-structured.…”

Improving Text Summarization Quality by Combining T5-Based Models and Convolutional Seq2Seq Models

“…In this study, we will combine the models used to summarize, namely the transformer or T5 model with the convolutional Seq2Seq model. Several related studies such as those conducted by Fendji (Fendji et al, 2021) conducted summaries on French Wikipedia documents by applying the T5 model, the T5 model has several advantages in generating text so many researchers use the T5 model in conducting summaries as did Chouikhi (Chouikhi & Alsuhaibani, 2022), Jung (Jung et al, 2022), and Quatra (La Quatra & Cagliero, 2022). In many studies in conducting abstract summaries, there are challenges such as unstructured text which must be changed with text preprocessing techniques such as those carried out by Widyassari (Widyassari et al, 2022) and Christian (Christian et al, 2016) utilizing text preprocessing techniques to be used for document summary processes.…”

DMSeqNet-mBART: A State-of-the-Art Adaptive-DropMessage Enhanced mBART Architecture for Superior Chinese Short News Text Summarization

“…To fully address these issues, it is therefore crucial to enhance the model architecture specifically for Chinese. Inspired by the mBART model's summary generation and BART-IT [9] model's success in Italian summarization, our team has optimized the mBART model for Chinese short news(less than 500 Chinese characters) text summarization, aiming to develop a robust model for this specific domain.…”