Abstractive Text Summarization: Enhancing Sequence-to-Sequence Models Using Word Sense Disambiguation and Semantic Content Generalization

Kouris, Panagiotis; Alexandridis, Georgios; Stafylopatis, Andreas

doi:10.1162/coli_a_00417

Cited by 23 publications

(11 citation statements)

References 64 publications

(120 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Second, combination of models also looks very promising for opinion mining as additional information on text entities have been proven to improve the quality of the summarization models [65]. One suggestion is to combine T5 and LongFormer with topicaware summarization models [66][67][68][69] that integrate topical information into sequential ones; this would be especially helpful for pools of comments with varying length.…”

Section: Future Workmentioning

confidence: 99%

Transformer-Based Abstractive Summarization for Reddit and Twitter: Single Posts vs. Comment Pools in Three Languages

2022

View full text Add to dashboard Cite

Abstractive summarization is a technique that allows for extracting condensed meanings from long texts, with a variety of potential practical applications. Nonetheless, today’s abstractive summarization research is limited to testing the models on various types of data, which brings only marginal improvements and does not lead to massive practical employment of the method. In particular, abstractive summarization is not used for social media research, where it would be very useful for opinion and topic mining due to the complications that social media data create for other methods of textual analysis. Of all social media, Reddit is most frequently used for testing new neural models of text summarization on large-scale datasets in English, without further testing on real-world smaller-size data in various languages or from various other platforms. Moreover, for social media, summarizing pools of texts (one-author posts, comment threads, discussion cascades, etc.) may bring crucial results relevant for social studies, which have not yet been tested. However, the existing methods of abstractive summarization are not fine-tuned for social media data and have next-to-never been applied to data from platforms beyond Reddit, nor for comments or non-English user texts. We address these research gaps by fine-tuning the newest Transformer-based neural network models LongFormer and T5 and testing them against BART, and on real-world data from Reddit, with improvements of up to 2%. Then, we apply the best model (fine-tuned T5) to pools of comments from Reddit and assess the similarity of post and comment summarizations. Further, to overcome the 500-token limitation of T5 for analyzing social media pools that are usually bigger, we apply LongFormer Large and T5 Large to pools of tweets from a large-scale discussion on the Charlie Hebdo massacre in three languages and prove that pool summarizations may be used for detecting micro-shifts in agendas of networked discussions. Our results show, however, that additional learning is definitely needed for German and French, as the results for these languages are non-satisfactory, and more fine-tuning is needed even in English for Twitter data. Thus, we show that a ‘one-for-all’ neural-network summarization model is still impossible to reach, while fine-tuning for platform affordances works well. We also show that fine-tuned T5 works best for small-scale social media data, but LongFormer is helpful for larger-scale pool summarizations.

show abstract

Section: Future Workmentioning

confidence: 99%

Transformer-Based Abstractive Summarization for Reddit and Twitter: Single Posts vs. Comment Pools in Three Languages

2022

View full text Add to dashboard Cite

show abstract

“…In an effort to provide a qualitative assessment for evaluating the factual consistency (Kryściński et al, 2019;Goodrich et al, 2019) of the predicted summaries, we extend the approach of (Kouris et al, 2021) computing the precision, recall and f β scores of factual consistency. More specifically, triplets, such as (subset, relation, Goodrich et al, 2019).…”

Section: Factual Consistencymentioning

confidence: 99%

“…Since factual consistency concerns the domain of automatic TS (Kryściński et al, 2019;Goodrich et al, 2019;Kouris et al, 2021), Tables 5 and 6 report the factual consistency (Section 6.2) of the experiments on Gigaword and CNN/DailyMail datasets, respectively. To compute the f β score, we set β = 0.264 and β = 0.158 for the Gigawornd and CNN/DailyMail datasets, respectively, as these values represent the fractions of the average length of a summary to the average length of a text in the test sets of the datasets (the usage of β coefficient is explained in detail in Section 6.2).…”

Section: Modelmentioning

confidence: 99%

Text summarization based on semantic graphs: An abstract meaning representation graph-to-text deep learning approach

Kouris

Alexandridis

Stafylopatis

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Nowadays, due to the constantly growing amount of textual information, automatic text summarization constitutes an important research area in natural language processing. In this work, we present a novel framework that combines semantic graph representations along with deep learning predictions to generate abstractive summaries of single documents, in an effort to utilize a semantic representation of the unstructured textual content in a machine-readable, structured, and concise manner. The overall framework is based on a well defined methodology for performing semantic graph parsing, graph construction, graph transformations for machine learning models and deep learning predictions. The employed semantic graph representation focuses on using the model of abstract meaning representation. Several combinations of graph construction and graph transformation methods are investigated to specify the most efficient of them for the machine learning models. Additionally, a range of deep learning architectures is examined, including a sequence-to-sequence attentive network, reinforcement learning, transformer-based architectures, and pre-trained neural language models. In this direction, a semantic graph representation of an original text is extracted, and then the present framework formulates the problem as a graph-to-summary learning problem to predict a summary of an original text. To the best of our knowledge, this formulation of graph-to-summary predictions in abstractive text summarization, without other intermediate steps in the machine learning phase, has not been presented in the relevant literature. Another important contribution is an extension of a metric for assessing the factual consistency of the generated summaries in an effort to provide a qualitative evaluation. To assess the framework, an extensive experimental procedure is presented that uses popular datasets to evaluate key aspects of the proposed approach. The obtained results exhibit promising performance, validating the robustness of the proposed framework.

show abstract

“…You et al [13] presented a transformer-based encoder-decoder architecture with an encoder-integrated focus-attention mechanism and a separate saliencyselection network that regulates the information flow from encoder to decoder. To predict generalised and resilient summaries, Kouris et al [8] uses a coping and coverage strategy in encoder-decoder based models, as well as reinforcement learning.…”

Section: Related Workmentioning

confidence: 99%

“…Because they don't provide any useful information to the user inquiry, these named entities were eliminated from the input text. • Motivated by the work of Kouris et al [8], we use a word sense disambiguator to provide additional meaning to ambiguous statements. For words that are too specialised to the issue, the disambiguator provides further information to the user question.…”

Section: Technical Data Cleanermentioning

confidence: 99%

Metadata Based Contextual Summarizer for Technical Conversations in Public Forums

Ranjan¹,

Govindan²,

Verma

2022

Towards a Knowledge-Aware AI

View full text Add to dashboard Cite

In recent years, the task of sequence to sequence based neural abstractive summarization has gained a lot of attention. Many novel strategies have been used to improve the saliency, human readability, and consistency of these models, resulting in high-quality summaries. However, because the majority of these pretrained models were trained on news datasets, they contain an inherent bias. One such bias is that most of these generated summaries originate from the start or end of the text, much like a news story might be summarised. Another issue we encountered while using these summarizers in our Technical discussion forums usecase was token recurrence, which resulted in lower ROUGE-precision scores. To overcome these issues, we present a unique approach that includes: a) An additional parameter to the loss function based on ROUGE-precision score that is optimised alongside categorical cross entropy loss. b) An adaptive loss function based on token repetition rate which is optimized along with the final loss so that the model may provide contextual summaries with less token repetition and successfully learn with the least training samples. c) To effectively contextualize this summarizer for technical forum discussion platforms, we added extra metadata indicator tokens to aid the model in learning latent features and dependencies in text segments with relevant metadata information. To avoid overfitting due to data scarcity, we test and verify all models on a hold-out dataset that was not part of the training or validation dataset. This paper discusses the various strategies we used and compares the performance of fine tuned models against baseline summarizers n the test dataset. By end-to-end training our models with these losses, we acquire substantially better ROUGE scores while being the most legible and relevant summary on the Technical forum dataset.

show abstract

Abstractive Text Summarization: Enhancing Sequence-to-Sequence Models Using Word Sense Disambiguation and Semantic Content Generalization

Cited by 23 publications

References 64 publications

Transformer-Based Abstractive Summarization for Reddit and Twitter: Single Posts vs. Comment Pools in Three Languages

Transformer-Based Abstractive Summarization for Reddit and Twitter: Single Posts vs. Comment Pools in Three Languages

Text summarization based on semantic graphs: An abstract meaning representation graph-to-text deep learning approach

Metadata Based Contextual Summarizer for Technical Conversations in Public Forums

Contact Info

Product

Resources

About