Monotonic alignments for summarization

Chung, Tong Lee; Liu, Yongbin; Xu, Bin

doi:10.1016/j.knosys.2019.105363

Cited by 9 publications

(7 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Their approach outperforms several other models on grapheme-to-phoneme conversion, transliteration, and morphological inflection. Monotonic attention has also improved tasks such as summarization (Chung et al, 2020) and morphological analysis (Hwang and Lee, 2020).…”

Section: Related Workmentioning

confidence: 99%

On Biasing Transformer Attention Towards Monotonicity

Rios

Amrhein

Aepli

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Many sequence-to-sequence tasks in natural language processing are roughly monotonic in the alignment between source and target sequence, and previous work has facilitated or enforced learning of monotonic attention behavior via specialized attention functions or pretraining. In this work, we introduce a monotonicity loss function that is compatible with standard attention mechanisms and test it on several sequence-to-sequence tasks: grapheme-to-phoneme conversion, morphological inflection, transliteration, and dialect normalization. Experiments show that we can achieve largely monotonic behavior. Performance is mixed, with larger gains on top of RNN baselines. General monotonicity does not benefit transformer multihead attention, however, we see isolated improvements when only a subset of heads is biased towards monotonic behavior.

show abstract

Section: Related Workmentioning

confidence: 99%

On Biasing Transformer Attention Towards Monotonicity

Rios

Amrhein

Aepli

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

show abstract

“…See et al suggested a coverage penalty to effectively learn the coverage mechanism [4]. Chung et al suggested mechanisms and penalties to point words in the same sequence as the input document and a word near the word that has already been selected [8].…”

Section: Related Workmentioning

confidence: 99%

“…We used the non-anonymized version of the dataset like other research [4][5][6][7][8][9][10][11]13], that is, without a named entity recognition, part-of-speech tagging, and so on. Each data element was space tokenized and changed to lowercase.…”

Section: Datasetmentioning

confidence: 99%

See 1 more Smart Citation

Document Summarization Based on Coverage with Noise Injection and Word Association

Kim

Lee

2020

Information

View full text Add to dashboard Cite

Automatic document summarization is a field of natural language processing that is rapidly improving with the development of end-to-end deep learning models. In this paper, we propose a novel summarization model that consists of three methods. The first is a coverage method based on noise injection that makes the attention mechanism select only important words by defining previous context information as noise. This alleviates the problem that the summarization model generates the same word sequence repeatedly. The second is a word association method to update the information of each word by comparing the information of the current step with the information of all previous decoding steps. According to following words, this catches a change in the meaning of the word that has been already decoded. The third is a method using a suppression loss function that explicitly minimizes the probabilities of non-answer words. The proposed summarization model showed good performance on some recall-oriented understudy for gisting evaluation (ROUGE) metrics compared to the state-of-the-art models in the CNN/Daily Mail summarization task, and the results were achieved with very few learning steps compared to the state-of-the-art models.

show abstract

“…If the original rules or dictionary are used, it will cause low recall. The current advanced approaches are data-driven, including machine learning methods [7] and deep learning methods, especially the Bidirectional Long Short-Term Memory and Conditional Random Field (Bi-LSTM-CRF) models [8][9][10][11], which have been successfully used, achieved better results, and represent some of the more commonly used methods. Despite the great success of the deep learning method, there are still some problems that have not been well resolved.…”

Section: Introductionmentioning

confidence: 99%

Improving the Named Entity Recognition of Chinese Electronic Medical Records by Combining Domain Dictionary and Rules

Chen

Ouyang

Liu

et al. 2020

IJERPH

Self Cite

View full text Add to dashboard Cite

Electronic medical records are an integral part of medical texts. Entity recognition of electronic medical records has triggered many studies that propose many entity extraction methods. In this paper, an entity extraction model is proposed to extract entities from Chinese Electronic Medical Records (CEMR). In the input layer of the model, we use word embedding and dictionary features embedding as input vectors, where word embedding consists of a character representation and a word representation. Then, the input vectors are fed to the bidirectional long short-term memory to capture contextual features. Finally, a conditional random field is employed to capture dependencies between neighboring tags. We performed experiments on body classification task, and the F1 values reached 90.65%. We also performed experiments on anatomic region recognition task, and the F1 values reached 93.89%. On both tasks, our model had higher performance than state-of-the-art models, such as Bi-LSTM-CRF, Bi-LSTM-Attention, and Vote. Through experiments, our model has a good effect when dealing with small frequency entities and unknown entities; with a small training dataset, our method showed 2–4% improvement on F1 value compared to the basic Bi-LSTM-CRF models. Additionally, on anatomic region recognition task, besides using our proposed entity extraction model, 12 rules we designed and domain dictionary were adopted. Then, in this task, the weighted F1 value of the three specific entities extraction reached 84.36%.

show abstract

Monotonic alignments for summarization

Cited by 9 publications

References 4 publications

On Biasing Transformer Attention Towards Monotonicity

On Biasing Transformer Attention Towards Monotonicity

Document Summarization Based on Coverage with Noise Injection and Word Association

Improving the Named Entity Recognition of Chinese Electronic Medical Records by Combining Domain Dictionary and Rules

Contact Info

Product

Resources

About