Learning Hierarchical Structures On-The-Fly with a Recurrent-Recursive Model for Sequences

Abstract

We propose a hierarchical model for sequential data that learns a tree on-thefly, i.e. while reading the sequence. In the model, a recurrent network adapts its structure and reuses recurrent weights in a recursive manner. This creates adaptive skip-connections that ease the learning of long-term dependencies. The tree structure can either be inferred without supervision through reinforcement learning, or learned in a supervised manner. We provide preliminary experiments in a novel Math Expression Evaluation (MEE) task, which is explicitly crafted to have a hierarchical tree structure that can be used to study the effectiveness of our model. Additionally, we test our model in a wellknown propositional logic and language modelling tasks. Experimental results show the potential of our approach. * Equal contribution (Ordering determined by coin flip).

Assessing Incrementality in Sequence-to-Sequence Models

“…Although this concept can be related to the prioritisation of information in the human visual cortex (Hassabis et al, 2017), it seems contrary to the incremental processing of information in a language context, as for instance recently shown empirically for the understanding of conjunctive generic sentences (Tessler et al, 2019). In machine learning, the idea of incrementality has already played a role in several problem statements, such as inferring the tree structure of a sentence (Jacob et al, 2018), parsing (Köhn and Menzel, 2014), or in other problems that are naturally equipped with time constraints like realtime neural machine translation (Neubig et al, 2017;Dalvi et al, 2018a), and speech recognition (Baumann et al, 2009;Jaitly et al, 2016;Graves, 2012). Other approaches try to encourage incremental behavior implictly by modifying the model architecture or the training objective: Guan et al (2018) introduce an encoder with an incremental self-attention scheme for story generation.…”

Modeling Hierarchical Structures with Continuous Recursive Neural Networks

“…We compare CRvNN with Tree-LSTM (Tai et al, 2015), Tree-Cell (Shen et al, 2019a) Tree-RNN (Bowman et al, 2015b), Tranformer (Vaswani et al, 2017), Universal Transformer (Dehghani et al, 2019), LSTM (Hochreiter & Schmidhuber, 1997), RRNet (Jacob et al, 2018), ON-LSTM (Shen et al, 2019b), Ordered Memory (Shen et al, 2019a) (see Table 2).…”

Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks

“…The recursive application of the same composition function is well suited for this task. We also include the result of RRNet (Jacob et al, 2018), which can induce the latent tree structure from downstream tasks. Note that the results may not be comparable, because the hyper-parameters for training were not provided.…”

“…years (Bowman et al, 2016;Yogatama et al, 2016;Shen et al, 2017;Jacob et al, 2018;Choi et al, 2018;Williams et al, 2018;Shi et al, 2018).…”