Neural Abstractive Text Summarization with Sequence-to-Sequence Models

Yang

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

et al. 2021

A well-known limitation in pretrain-finetune paradigm lies in its inflexibility caused by the one-size-fits-all vocabulary. This potentially weakens the effect when applying pretrained models into natural language generation (NLG) tasks, especially for the subword distributions between upstream and downstream tasks with significant discrepancy. Towards approaching this problem, we extend the vanilla pretrain-finetune pipeline with an extra embedding transfer step. Specifically, a plug-and-play embedding generator is introduced to produce the representation of any input token, according to pre-trained embeddings of its morphologically similar ones. Thus, embeddings of mismatch tokens in downstream tasks can also be efficiently initialized. We conduct experiments on a variety of NLG tasks under the pretrain-finetune fashion. Experimental results and extensive analyses show that the proposed strategy offers us opportunities to feel free to transfer the vocabulary, leading to more efficient and better performed downstream NLG models. 1

“…In the future, we plan to investigate its effectiveness on other generation tasks, such as code generation (Jiang et al, 2021;, summarization (Shi et al, 2021) and so on.…”

Section: Discussionmentioning

confidence: 99%

Bridging Subword Gaps in Pretrain-Finetune Paradigm for Natural Language Generation

Yang

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

“…We employ an encoder-decoder architecture for the problem defined above, which is similar to most Seq2Seq models in machine translation (Vaswani et al, 2017;Zhang et al, 2018), automatic text summarization (Song et al, 2019;Shi et al, 2021), and speech recognition (Tüske et al, 2019;Hannun et al, 2019) from a high-level perspective. → y i and ← − y i ) respectively.…”

Section: The Proposed Approachmentioning

confidence: 99%

Capturing Event Argument Interaction via A Bi-Directional Entity-Level Recurrent Decoder

Xi¹,

Ye²,

Zhang³

et al. 2021

Capturing interactions among event arguments is an essential step towards robust event argument extraction (EAE). However, existing efforts in this direction suffer from two limitations: 1) The argument role type information of contextual entities is mainly utilized as training signals, ignoring the potential merits of directly adopting it as semantically rich input features; 2) The argument-level sequential semantics, which implies the overall distribution pattern of argument roles over an event mention, is not well characterized. To tackle the above two bottlenecks, we formalize EAE as a Seq2Seq-like learning problem for the first time, where a sentence with a specific event trigger is mapped to a sequence of event argument roles. A neural architecture with a novel Bi-directional Entity-level Recurrent Decoder (BERD) is proposed to generate argument roles by incorporating contextual entities' argument role predictions, like a wordby-word text generation process, thereby distinguishing implicit argument distribution patterns within an event more accurately.

“…The research in Automatic Text Summarization is enriched with many surveys that have been conducted and published in the past years. The survey conducted by [3] and [4] are quite extensive while those conducted by [1] and [5]- [7] are centered towards extractive and abstractive summarization. This research study tends to survey the scientific literature to obtain information and knowledge about the recent research in automatic text summarization specifically abstractive summarization based on neural networks to have an understanding and familiarity with the design of stateof-the-art models in this realm.…”

Section: Introductionmentioning

confidence: 99%

A Survey of the State-of-the-Art Models in Neural Abstractive Text Summarization

2021

Dealing with vast amounts of textual data requires the use of efficient systems. Automatic summarization systems are capable of addressing this issue. Therefore, it becomes highly essential to work on the design of existing automatic summarization systems and innovate them to make them capable of meeting the demands of continuously increasing data, based on user needs. This study tends to survey the scientific literature to obtain information and knowledge about the recent research in automatic text summarization specifically abstractive summarization based on neural networks. A review of various neural networks based abstractive summarization models have been presented. The proposed conceptual framework includes five key elements identified as encoder-decoder architecture, mechanisms, training strategies and optimization algorithms, dataset, and evaluation metric. A description of these elements is also included in this article. The purpose of this research is to provide an overall understanding and familiarity with the elements of recent neural networks based abstractive text summarization models with an up-to-date review as well as to render an awareness of the challenges and issues with these systems. Analysis has been performed qualitatively with the help of a concept matrix indicating common trends in the design of recent neural abstractive summarization systems. Models employing a transformer-based encoder-decoder architecture are found to be the new state-of-the-art. Based on the knowledge acquired from the survey, this article suggests the use of pre-trained language models in complement with neural network architecture for abstractive summarization task.