Recurrent Neural Network Grammars

Abstract

This is modified version of a paper originally published at NAACL 2016 that contains a corrigendum at the end, with improved results after fixing an implementation bug in the RNNG composition function. AbstractWe introduce recurrent neural network grammars, probabilistic models of sentences with explicit phrase structure. We explain efficient inference procedures that allow application to both parsing and language modeling. Experiments show that they provide better parsing in English than any single previously published supervised generative model and better language modeling than state-of-the-art sequential RNNs in English and Chinese 1 .

Improving Feature-Rich Transition-Based Constituent Parsing Using Recurrent Neural Networks

“…As reported in [9], reranking with the generative RNN models deliver gains over end-to-end discriminative RNN models, our framework thereby has the potential to advance the reranking models [9], [11] by integrating these generative RNN models into decoding † , which remains a future work. On the German dataset, particularly, our parser outperforms other state-of-the-art parsers both when using auto-labelled POS tags and gold tags.…”

“…For example, [7] proposed an end-to-end RNN parser whose performance is comparable to the stateof-the-art feature-rich parsers, and [9] proposed a variant RNN model for parsing. Unlike their pure neural network models, ours is a hybrid model consisting of neural networks and feature-rich models, which can combine their merits.…”

“…On the other hand, recently recurrent neural networks (RNNs) have been applied successfully to many NLP tasks, particularly to constituent parsing [6]- [9]. By encoding the unbounded history into a dense representation, RNNs are able to capture the long-term dependencies and thus make reliable parsing decisions.…”

“…The proposed framework is general to be applied on top of other RNN models in [7], [9], [11], although the sequenceto-sequence RNN model is instantiated in this paper.…”

Adaptive Importance Sampling from Finite State Automata

“…The first assumption is violated when e.g. non-local features (Huang, 2008) are used to define probabilities or when probabilities are defined by recurrent neural nets that use hidden states derived from whole subtrees (Socher et al, 2013;Dyer et al, 2016). The second assumption is violated when e.g.…”