The Importance of Being Recurrent for Modeling Hierarchical Structure

Abstract

Recent work has shown that recurrent neural networks (RNNs) can implicitly capture and exploit hierarchical information when trained to solve common natural language processing tasks (Blevins et al., 2018) such as language modeling (Linzen et al., 2016;Gulordava et al., 2018) and neural machine translation (Shi et al., 2016). In contrast, the ability to model structured data with non-recurrent neural networks has received little attention despite their success in many NLP tasks (Gehring et al., 2017;Vaswani et al., 2017). In this work, we compare the two architectures-recurrent versus non-recurrent-with respect to their ability to model hierarchical structure and find that recurrency is indeed important for this purpose. The code and data used in our experiments is available at https://github.com/ ketranm/fan_vs_rnn

Deep Contextualized Word Embeddings in Transition-Based and Graph-Based Dependency Parsing - A Tale of Two Parsers Revisited

“…For ELMo there is still a discernible difference for dependencies longer than 5, but for BERT the two curves are almost indistinguishable throughout the whole range. This could be related to the aforementioned intuition that a Transformer captures long dependencies more effectively than a BiLSTM (see Tran et al (2018) for contrary observations, albeit for different tasks). The overall trends for both baseline and enhanced models are quite consistent across languages, although with large variations in accuracy levels.…”

Revealing the Dark Secrets of BERT

“…Jawahar et al (2019) extended this work to using multiple layers and tasks, supporting the claim that BERT's intermediate layers capture rich linguistic information. On the other hand, Tran et al (2018) concluded that LSTMs generalize to longer sequences better, and are more robust with respect to agreement distractors, compared to Transformers. Liu et al (2019) investigated the transferability of contextualized word representations to a number of probing tasks requiring linguistic knowledge.…”

Style-Aware Neural Model with Application in Authorship Attribution

“…The primary reason for adopting recurrent architecture for sentence-encoder is because recurrent neural networks have been shown to be essential for capturing the underlying hierarchical structure of sequential data [14]. By adopting this approach sentence-encoder is able to encode how sentences are structured in a document.…”

“…For sentence-level encoder, we employ an attention-based recurrent neural network to capture the struc-tural patterns of sentences in the document. The primary reason for adopting recurrent architecture for sentence-encoder is because recurrent neural networks have been shown to be essential for capturing the underlying hierarchical structure of sequential data [14]. Hence, sentence-encoder in the proposed model is expected to capture the structural information of documents.…”