A Survey of Abstractive Text Summarization Utilising Pretrained Language Models

Syed, Ayesha Ayub; Gaol, Ford Lumban; Boediman, Alfred; Matsuo, Tokuro; Budiharto, Widodo

doi:10.1007/978-3-031-21743-2_42

Cited by 4 publications

(6 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Addressing these challenges necessitates a more balanced approach to learning. One potential avenue is the incorporation of regularization techniques [72]. Regularization, in essence, adds a penalty to the loss function, discouraging the model from fitting too closely to every data point and, in turn, mitigating overfitting.…”

Section: ) Explore Noisementioning

confidence: 99%

Automatic Extractive Summarization using GAN Boosted by DistilBERT Word Embedding and Transductive Learning

Li,

ZHANG

2023

IJACSA

View full text Add to dashboard Cite

Text summarization is crucial in diverse fields such as engineering and healthcare, greatly enhancing time and cost efficiency. This study introduces an innovative extractive text summarization approach utilizing a Generative Adversarial Network (GAN), Transductive Long Short-Term Memory (TLSTM), and DistilBERT word embedding. DistilBERT, a streamlined BERT variant, offers significant size reduction (approximately 40%), while maintaining 97% of language comprehension capabilities and achieving a 60% speed increase. These benefits are realized through knowledge distillation during pre-training. Our methodology uses GANs, consisting of the generator and discriminator networks, built primarily using TLSTM -an expert at decoding temporal nuances in timeseries prediction. For more effective model fitting, transductive learning is employed, assigning higher weights to samples nearer to the test point. The generator evaluates the probability of each sentence for inclusion in the summary, and the discriminator critically examines the generated summary. This reciprocal relationship fosters a dynamic iterative process, generating toptier summaries. To train the discriminator efficiently, a unique loss function is proposed, incorporating multiple factors such as the generator's output, actual document summaries, and artificially created summaries. This strategy motivates the generator to experiment with diverse sentence combinations, generating summaries that meet high-quality and coherence standards. Our model's effectiveness was tested on the widely accepted CNN/Daily Mail dataset, a benchmark for summarization tasks. According to the ROUGE metric, our experiments demonstrate that our model outperforms existing models in terms of summarization quality and efficiency.

show abstract

Section: ) Explore Noisementioning

confidence: 99%

Automatic Extractive Summarization using GAN Boosted by DistilBERT Word Embedding and Transductive Learning

Li,

ZHANG

2023

IJACSA

View full text Add to dashboard Cite

show abstract

“… The data is particularly useful for training deep transfer learning models for title generation, abstractive summarization [ [1] , [5] ], and sentiment analysis of airline reviews. The title generation data is applicable for domain adaptive training [ [2] , [3] ] of pretrained language models to improve performance on the target language generation tasks for the airline reviews domain.…”

Section: Value Of the Datamentioning

confidence: 99%

“…The data is particularly useful for training deep transfer learning models for title generation, abstractive summarization [ [1] , [5] ], and sentiment analysis of airline reviews.…”

Section: Value Of the Datamentioning

confidence: 99%

“…The dataset for sentiment analysis is multipurpose having columns like user rating and recommendation value, that can be used for statistical analysis like finding correlation between these data items as well as for other Natural Language Processing (NLP) tasks like predicting rating or recommendation value from the customer reviews. The datasets can be extended using various data augmentation techniques [4 , 5] . Moreover, the datasets are related and can be collectively used to develop a multi-task learning model [6] for better learning efficiency and improved performance.…”

mentioning

confidence: 99%

See 1 more Smart Citation

A data package for abstractive opinion summarization, title generation, and rating-based sentiment prediction for airline reviews

Syed,

Gaol,

Boediman

et al. 2023

Data in Brief

View full text Add to dashboard Cite

“…Moreover, open-source models offer high adaptability and can be readily fine-tuned on domain-specific datasets with minimal effort. Furthermore, previous studies have demonstrated that both BART and T5 can generate summaries of comparable quality to those produced by smaller GPT-3 models [9][10][11].…”

Section: Introductionmentioning

confidence: 99%

Using Pre-Trained Language Models for Abstractive DBPEDIA Summarization: A Comparative Study

Zahera,

Vitiugin,

Sherif

et al. 2023

Knowledge Graphs: Semantics, Machine Learning, and Languages

View full text Add to dashboard Cite

Purpose: This study addresses the limitations of current short abstracts of DBPEDIA entities, which often lack a comprehensive overview due to their creating method (i.e., selecting the first two-three sentences from the full DBPEDIA abstracts). Methodology: We leverage pre-trained language models to generate abstractive summaries of DBPEDIA abstracts in six languages (English, French, German, Italian, Spanish, and Dutch). We performed several experiments to assess the quality of generated summaries by language models. In particular, we evaluated the generated summaries using human judgments and automated metrics (Self-ROUGE and BERTScore). Additionally, we studied the correlation between human judgments and automated metrics in evaluating the generated summaries under different aspects: informativeness, coherence, conciseness, and fluency. Findings: Pre-trained language models generate summaries more concise and informative than existing short abstracts. Specifically, BART-based models effectively overcome the limitations of DBPEDIA short abstracts, especially for longer ones. Moreover, we show that BERTScore and ROUGE-1 are reliable metrics for assessing the informativeness and coherence of the generated summaries with respect to the full DBPEDIA abstracts. We also find a negative correlation between conciseness and human ratings. Furthermore, fluency evaluation remains challenging without human judgment. Value: This study has significant implications for various applications in machine learning and natural language processing that rely on DBPEDIA resources. By providing succinct and comprehensive summaries, our approach enhances the quality of DBPEDIA abstracts and contributes to the semantic web community.

show abstract

A Survey of Abstractive Text Summarization Utilising Pretrained Language Models

Cited by 4 publications

References 18 publications

Automatic Extractive Summarization using GAN Boosted by DistilBERT Word Embedding and Transductive Learning

Automatic Extractive Summarization using GAN Boosted by DistilBERT Word Embedding and Transductive Learning

A data package for abstractive opinion summarization, title generation, and rating-based sentiment prediction for airline reviews

Using Pre-Trained Language Models for Abstractive DBPEDIA Summarization: A Comparative Study

Contact Info

Product

Resources

About