Scalable Syntax-Aware Language Models Using Knowledge Distillation

Abstract

Prior work has shown that, on small amounts of training data, syntactic neural language models learn structurally sensitive generalisations more successfully than sequential language models. However, their computational complexity renders scaling difficult, and it remains an open question whether structural biases are still necessary when sequential models have access to ever larger amounts of training data. To answer this question, we introduce an efficient knowledge distillation (KD) technique that transfers knowledge from a syntactic language model trained on a small corpus to an LSTM language model, hence enabling the LSTM to develop a more structurally sensitive representation of the larger training data it learns from. On targeted syntactic evaluations, we find that, while sequential LSTMs perform much better than previously reported, our proposed technique substantially improves on this baseline, yielding a new state of the art. Our findings and analysis affirm the importance of structural biases, even in models that learn from large amounts of data.

An Analysis of the Utility of Explicit Negative Examples to Improve the Syntactic Abilities of Neural Language Models

“…Another relevant work on the capacity of LSTM-LMs is Kuncoro et al (2019), which shows that by distilling from syntactic LMs (Dyer et al, 2016), LSTM-LMs can improve their robustness on various agreement phenomena. We show that our LMs with the margin loss outperform theirs in most of the aspects, further strengthening the argument about a stronger capacity of LSTM-LMs.…”

“…Training data Following the practice, we train LMs on the dataset not directly relevant to the test set. Throughout the paper, we use an English Wikipedia corpus assembled by Gulordava et al (2018), which has been used as training data for the present task (Marvin and Linzen, 2018;Kuncoro et al, 2019), consisting of 80M/10M/10M tokens for training/dev/test sets. It is tokenized and rare words are replaced by a single unknown token, amounting to the vocabulary size of 50,000.…”

“…Deviating from some prior work (Marvin and Linzen, 2018;van Schijndel et al, 2019), we train LMs at sentence level as in sequence-tosequence models (Sutskever et al, 2014). This setting has been employed in some previous work (Kuncoro et al, 2018(Kuncoro et al, , 2019. 2 Parameters are optimized by SGD.…”

Mimic and Conquer: Heterogeneous Tree Structure Distillation for Syntactic NLP

“…Our work is also closely related to Kuncoro et al, (2019), who distill syntactic structure knowledge to a student LSTM model. The difference lies in that they focus on transferring tree knowledge from syntax-aware language model for achieving scalable unsupervised syntax induction, while we aim at integrating heterogeneous syntax for improving downstream tasks.…”

“…Sequential models have been proven effective on encoding syntactic tree information (Shen et al, 2018;Kuncoro et al, 2019). We set the goal of KD as simultaneously distilling heterogeneous structures from tree encoder teachers into a LSTM student model.…”