Higher-Order Syntactic Attention Network for Longer Sentence Compression

Abstract

Sentence compression methods based on LSTM can generate fluent compressed sentences. However, the performance of these methods is significantly degraded when compressing long sentences since it does not explicitly handle syntactic features. To solve this problem, we propose a higher-order syntactic attention network (HiSAN) that can handle higher-order dependency features as an attention distribution on LSTM hidden states. Furthermore, to avoid the influence of incorrect parse results, we train HiSAN by maximizing the probability of a correct output together with the attention distribution. Experiments on the Google sentence compression dataset show that our method achieved the best performance in terms of F 1 as well as ROUGE-1,2 and L scores, 83.2, 82.9, 75.8 and 82.7, respectively. In subjective evaluations, HiSAN outperformed baseline methods in both readability and informativeness.

Data-efficient Neural Text Compression with Interactive Learning

“…For our experiments, we use the large Google News text compression corpus 3 by Filippova and Altun (2013), which contains 250k automatically extracted the deletion-based compressions from aligned headlines and first sentences of news articles. Recent studies on text compression have extensively used this dataset (e.g., Zhao et al, 2018;Kamigaito et al, 2018). We carry out in-domain active learning experiments on the Google News compression corpus.…”

“…Neural sequence-to-sequence (Seq2Seq) models have shown remarkable success in many areas of natural language processing and specifically in natural language generation tasks, including text compression (Rush et al, 2015;Filippova et al, 2015;Yu et al, 2018;Kamigaito et al, 2018). Despite their success, Seq2Seq models have a major drawback, as they require huge parallel cor-pora with pairs of source and compressed text to be able to learn the parameters for the model.…”

Abstractive Sentence Compression with Event Attention

“…Chopra et al proposed a convolutional attention-based model [31] in which the attention helps to understand a source sentence by capturing its context. On the other hand, Kamigatio et al incorporated a higher-order syntactic attention into a sequence-to-sequence model [8]. The attention is computed with a chain of dependency relations in the dependency graph of a source sentence.…”

“…There have been many studies about sentence compression and the studies are separated into two approaches: a deletion-based approach and an abstractive approach. The deletion-based approach generates a target sentence by removing unimportant and unnecessary words from the source sentence [3][4][5][6][7][8]. That is, a target compressed sentence is a subsequence of a source sentence.…”

“…In addition, its attention mechanism often fails to focus on important content of a source sentence. In particular, when a source sentence is long, the attention failure gets severer [8,16,17].…”