Larger-Context Language Modelling with Recurrent Neural Network

Wang, Tian; Cho, Kyunghyun

doi:10.18653/v1/p16-1125

Cited by 50 publications

(44 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Section: Related Workmentioning

confidence: 99%

“…Comparison to Exploiting Auxiliary Contexts in Language Modeling: A thread of work in language modeling (LM) attempts to exploit auxiliary sentence-level or document-level context in an RNN LM (Mikolov and Zweig, 2012;Ji et al, 2015;Wang and Cho, 2016). Independent of our work, Wang and Cho (2016) propose "early fusion" models of RNNs where additional information from an intersentence context is "fused" with the input to the RNN. Closely related to Wang and Cho (2016), our approach aims to dynamically control the contributions of required source and target contexts for machine translation, while theirs focuses on integrating auxiliary corpus-level contexts for language modelling to better approximate the corpus-level probability.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Context Gates for Neural Machine Translation

Liu

et al. 2017

TACL

View full text Add to dashboard Cite

In neural machine translation (NMT), generation of a target word depends on both source and target contexts. We find that source contexts have a direct impact on the adequacy of a translation while target contexts affect the fluency. Intuitively, generation of a content word should rely more on the source context and generation of a functional word should rely more on the target context. Due to the lack of effective control over the influence from source and target contexts, conventional NMT tends to yield fluent but inadequate translations. To address this problem, we propose context gates which dynamically control the ratios at which source and target contexts contribute to the generation of target words. In this way, we can enhance both the adequacy and fluency of NMT with more careful control of the information flow from contexts. Experiments show that our approach significantly improves upon a standard attentionbased NMT system by +2.3 BLEU points.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Context Gates for Neural Machine Translation

Liu

et al. 2017

TACL

View full text Add to dashboard Cite

show abstract

“…A larger context language model that incorporates context from preceding sentences (Wang and Cho, 2016), by treating the preceding sentence as a bag of words, and using an attentional mechanism when predicting the next word. An additional hyper-parameter in lclm is the number of preceeding sentences to incorporate, which we tune based on a development set (to 4 sentences in each case).…”

Section: Lclmmentioning

confidence: 99%

Topically Driven Neural Language Model

Lau¹,

Baldwin²,

Cohn³

2017

Proceedings of the 55th Annual Meeting of the Association For Computational Linguistics (Volume 1: Long Papers)

View full text Add to dashboard Cite

Language models are typically applied at the sentence level, without access to the broader document context. We present a neural language model that incorporates document context in the form of a topic model-like architecture, thus providing a succinct representation of the broader document context outside of the current sentence. Experiments over a range of datasets demonstrate that our model outperforms a pure sentence-based model in terms of language model perplexity, and leads to topics that are potentially more coherent than those produced by a standard LDA topic model. Our model also has the ability to generate related sentences for a topic, providing another way to interpret topics.

show abstract

“…Finally, we use late fusion (Wang and Cho, 2015) to combine the output of the attention mechanism with the current position in the B-RNN without interfering with its memory.…”

Section: The Neural Network Modelmentioning

confidence: 99%

Deep Learning for Punctuation Restoration in Medical Reports

Salloum¹,

Finley²,

Edwards³

et al. 2017

BioNLP 2017

View full text Add to dashboard Cite

In clinical dictation, speakers try to be as concise as possible to save time, often resulting in utterances without explicit punctuation commands. Since the end product of a dictated report, e.g. an out-patient letter, does require correct orthography, including exact punctuation, the latter need to be restored, preferably by automated means. This paper describes a method for punctuation restoration based on a stateof-the-art stack of NLP and machine learning techniques including B-RNNs with an attention mechanism and late fusion, as well as a feature extraction technique tailored to the processing of medical terminology using a novel vocabulary reduction model. To the best of our knowledge, the resulting performance is superior to that reported in prior art on similar tasks.

show abstract

Larger-Context Language Modelling with Recurrent Neural Network

Cited by 50 publications

References 35 publications

Context Gates for Neural Machine Translation

Context Gates for Neural Machine Translation

Topically Driven Neural Language Model

Deep Learning for Punctuation Restoration in Medical Reports

Contact Info

Product

Resources

About