Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks

Abstract

Natural language is hierarchically structured: smaller units (e.g., phrases) are nested within larger units (e.g., clauses). When a larger constituent ends, all of the smaller constituents that are nested within it must also be closed. While the standard LSTM architecture allows different neurons to track information at different time scales, it does not have an explicit bias towards modeling a hierarchy of constituents. This paper proposes to add such an inductive bias by ordering the neurons; a vector of master input and forget gates ensures that when a given neuron is updated, all the neurons that follow it in the ordering are also updated. Our novel recurrent architecture, ordered neurons LSTM (ON-LSTM), achieves good performance on four different tasks: language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference 1 . * Equal contribution. {yi-kang.shen,jing.shan.shawn.tan}@umontreal.ca.

Ordered Memory

“…In this paper, we introduce the Ordered Memory architecture. The model is conceptually close to previous stack-augmented RNNs, but with two important differences: 1) we replace the pop and push operations with a new writing and erasing mechanism inspired by Ordered Neurons (Shen et al, 2018); 2) we also introduce a new Gated Recursive Cell to compose lower level representations into higher level one. On the logical inference and ListOps tasks, we show that the model learns the proper tree structures required to solve them.…”

“…Ordered Memory is implemented following the principles introduced in Ordered Neurons (Shen et al, 2018). Our model is related to ON-LSTM in several aspects: 1) The memory slots are similar to the chunks in ON-LSTM, when a higher ranking memory slot is forgotten/updated, all lower ranking memory slots should likewise be forgotten/updated; 2) ON-LSTM uses the monotonically non-decreasing master forget gate to preserve long-term information while erasing short term information, the OM model uses the cumulative probability − → π t ; 3) Similarly, the master input gate used by ON-LSTM to control the writing of new information into the memory is replaced with the reversed cumulative probability ← − π t in the OM model.…”

“…We use a multi-layer perceptron (MLP) with (h 1 , h 2 , h 1 • h 2 , |h 1 −h 2 |) as input, where h 1 and h 2 are the M N T of their respective input sequences. We compare our model with LSTM, RRNet (Jacob et al, 2018), ON-LSTM (Shen et al, 2018), Tranformer (Vaswani et al, 2017), Universal Transformer (Dehghani et al, 2018), TreeLSTM (Tai et al, 2015), TreeRNN (Bowman et al, 2015), and TreeCell. We used the same hidden state size for our model and baselines for proper comparison.…”

“…In recent years, many attempts have been made in this direction using neural network architectures (Grefenstette et al, 2015;Bowman et al, 2016;Williams et al, 2018;Shen et al, 2018;. These models typically augment a recurrent neural network with a stack and a buffer which operate in a similar way to how a shift-reduce parser builds a parse-tree.…”

Can RNNs learn Recursive Nested Subject-Verb Agreements?

“…Ordered-Neurons LSTMs (ON-LSTMs) Based on the intuition that larger constituents contain information that changes more slowly across the sentence, Shen et al [2018] suggested a variant of LSTMs, called Ordered-Neurons LSTMs, which imposes a hierarchical bias on the cell-updating mechanism. Given the hierarchical nature of our data, we expected ON-LSTMs to perform well on the number-agreements tasks.…”