Pointing the Unknown Words

Abstract

The problem of rare and unknown words is an important issue that can potentially effect the performance of many NLP systems, including traditional count-based and deep learning models. We propose a novel way to deal with the rare and unseen words for the neural network models using attention. Our model uses two softmax layers in order to predict the next word in conditional language models: one predicts the location of a word in the source sentence, and the other predicts a word in the shortlist vocabulary. At each timestep, the decision of which softmax layer to use is adaptively made by an MLP which is conditioned on the context. We motivate this work from a psychological evidence that humans naturally have a tendency to point towards objects in the context or the environment when the name of an object is not known. Using our proposed model, we observe improvements on two tasks, neural machine translation on the Europarl English to French parallel corpora and text summarization on the Gigaword dataset.

Towards Problem Solving Agents that Communicate and Learn

“…We accomplish this by using an attentionbased copying mechanism (Jia and Liang, 2016;Gulcehre et al, 2016;Gu et al, 2016). At every time step, the decoder may either output a token from the training vocabulary or copy a word from the input sentence.…”

Proceedings of the 2nd Workshop on Representation Learning for NLP

“…Particularly, we base our model on the attention mechanism of and the pointer-softmax copying mechanism of Gulcehre et al (2016). In question generation, we can condition our encoder on two different sources of information (compared to the single source in neural machine translation (NMT)): a document that the question should be about and an answer that should fit the generated question.…”

“…When formulating questions based on documents, it is common to refer to phrases and entities that appear directly in the text. We therefore incorporate into our decoder a mechanism for copying relevant words from D. We use the pointer-softmax formulation (Gulcehre et al, 2016), which has two output layers: the shortlist softmax and the location softmax. The shortlist softmax places a distribution over words in a predefined output vocabulary.…”

“…It is computed by an MLP with two hidden layers using tanh units (Gulcehre et al, 2016). Similarly to the computation of the shortlist softmax, the switching network takes s t , v t and y t−1 as inputs.…”

Cutting-off Redundant Repeating Generations for Neural Abstractive Summarization

“…Thus, we expect to decisively prohibit excessive generation. Finally, we evaluate the effectiveness of our method on well-studied ABS benchmark data provided by Rush et al (2015), and evaluated in (Chopra et al, 2016;Nallapati et al, 2016b;Kikuchi et al, 2016;Takase et al, 2016;Ayana et al, 2016;Gulcehre et al, 2016).…”