Formal Language Theory Meets Modern NLP

Merrill, William

doi:10.48550/arxiv.2102.10094

Cited by 2 publications

(3 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…No We vary m, the dimension of the hidden state, in the range [3,32], used RMSProp optimizer [49] with the smoothing constant α = 0.99 and varied the learning rate in the range [10 −2 , 10 −3 ]. For each language we train models corresponding to each language for 100 epochs and a batch size of 32.…”

Section: φ R Smentioning

confidence: 99%

“…RNNs can be viewed as dynamical systems and many works have used this viewpoint to study RNNs, e.g., [22,23,24,25]. Other related work includes relation to kernel methods, e.g., [26,27,28], linear RNNs [29], saturated RNNs [30,31,32], and echo state networks [33,34]. Several other works talk about the expressive power of the novel sequence to sequence models Transformers [35,36].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Learning and Generalization in RNNs

Panigrahi¹,

Goyal²

2021

Preprint

View full text Add to dashboard Cite

Simple recurrent neural networks (RNNs) and their more advanced cousins LSTMs etc. have been very successful in sequence modeling. Their theoretical understanding, however, is lacking and has not kept pace with the progress for feedforward networks, where a reasonably complete understanding in the special case of highly overparametrized one-hidden-layer networks has emerged. In this paper, we make progress towards remedying this situation by proving that RNNs can learn functions of sequences. In contrast to the previous work that could only deal with functions of sequences that are sums of functions of individual tokens in the sequence, we allow general functions. Conceptually and technically, we introduce new ideas which enable us to extract information from the hidden state of the RNN in our proofs-addressing a crucial weakness in previous work. We illustrate our results on some regular language recognition problems.

show abstract

Section: φ R Smentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Learning and Generalization in RNNs

Panigrahi¹,

Goyal²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…This memory store is a vector of hidden units h called the hidden state. Connections between RNNs and automata have been studied extensively (Chen et al, 2018;Peng et al, 2018;Merrill, 2019;Merrill et al, 2020;Merrill, 2021).…”

Section: Stack Rnnsmentioning

confidence: 99%

Dance preservation archives

Dusell¹

View full text Add to dashboard Cite

Human language is full of compositional syntactic structures, and although neural networks have contributed to groundbreaking improvements in computer systems that process language, widely-used neural network architectures still exhibit limitations in their ability to process syntax. To address this issue, prior work has proposed adding stack data structures to neural networks, drawing inspiration from theoretical connections between syntax and stacks. However, these methods employ deterministic stacks that are designed to track one parse at a time, whereas syntactic ambiguity, which requires a nondeterministic stack to parse, is extremely common in language. In this dissertation, we

show abstract

Formal Language Theory Meets Modern NLP

Cited by 2 publications

References 14 publications

Learning and Generalization in RNNs

Learning and Generalization in RNNs

Dance preservation archives

Contact Info

Product

Resources

About