WATS-SMS: A T5-Based French Wikipedia Abstractive Text Summarizer for SMS

Abstract

Text summarization remains a challenging task in the natural language processing field despite the plethora of applications in enterprises and daily life. One of the common use cases is the summarization of web pages which has the potential to provide an overview of web pages to devices with limited features. In fact, despite the increasing penetration rate of mobile devices in rural areas, the bulk of those devices offer limited features in addition to the fact that these areas are covered with limited connectivity such as the GSM network. Summarizing web pages into SMS becomes, therefore, an important task to provide information to limited devices. This work introduces WATS-SMS, a T5-based French Wikipedia Abstractive Text Summarizer for SMS. It is built through a transfer learning approach. The T5 English pre-trained model is used to generate a French text summarization model by retraining the model on 25,000 Wikipedia pages then compared with different approaches in the literature. The objective is twofold: (1) to check the assumption made in the literature that abstractive models provide better results compared to extractive ones; and (2) to evaluate the performance of our model compared to other existing abstractive models. A score based on ROUGE metrics gave us a value of 52% for articles with length up to 500 characters against 34.2% for transformer-ED and 12.7% for seq-2seq-attention; and a value of 77% for articles with larger size against 37% for transformers-DMCA. Moreover, an architecture including a software SMS-gateway has been developed to allow owners of mobile devices with limited features to send requests and to receive summaries through the GSM network.

Improving Text Summarization Quality by Combining T5-Based Models and Convolutional Seq2Seq Models

“…In this study, we will combine the models used to summarize, namely the transformer or T5 model with the convolutional Seq2Seq model. Several related studies such as those conducted by Fendji (Fendji et al, 2021) conducted summaries on French Wikipedia documents by applying the T5 model, the T5 model has several advantages in generating text so many researchers use the T5 model in conducting summaries as did Chouikhi (Chouikhi & Alsuhaibani, 2022), Jung (Jung et al, 2022), and Quatra (La Quatra & Cagliero, 2022). In many studies in conducting abstract summaries, there are challenges such as unstructured text which must be changed with text preprocessing techniques such as those carried out by Widyassari (Widyassari et al, 2022) and Christian (Christian et al, 2016) utilizing text preprocessing techniques to be used for document summary processes.…”

“…For unstructured data, it is necessary to carry out text processing which aims to improve the quality of the text so that it can be used as a dataset for text summary research. The T5 model has an architecture that can be used in developing text summaries (Ranganathan & Abuka, 2022) (Ramesh et al, 2022) while the convolutional Seq2Seq model has parameter tuning that can improve the quality of summaries (Fendji et al, 2021). Many related studies have implemented text summary models, text classification, and clustering as shown in Figure 1.…”

Enhancing Text Summarization with a T5 Model and Bayesian Optimization

“…In carrying out text summary tasks there are many techniques that can be used, one of which is using the most popular models such as the Transformer-based Text-to-Text Transfer Transformer or the T5 model which has advantages in producing good text summaries. However, several studies conducted by Mastropaolo et al [21], Fendji et al [22], and Blekanov et al [23] revealed that the T5 model still has room to improve performance in the process of carrying out text summary tasks, this study will use Bayesian optimization techniques that will perform a task in increasing the performance of the T5 model. In the Bayesian optimization task, we will use Bayesian probability theory for an iterative model so that it can have the advantage of updating initial knowledge.…”