An Analysis of the Utility of Explicit Negative Examples to Improve the Syntactic Abilities of Neural Language Models

Abstract

We explore the utilities of explicit negative examples in training neural language models. Negative examples here are incorrect words in a sentence, such as barks in *The dogs barks. Neural language models are commonly trained only on positive examples, a set of sentences in the training data, but recent studies suggest that the models trained in this way are not capable of robustly handling complex syntactic constructions, such as long-distance agreement. In this paper, we first demonstrate that appropriately using negative examples about particular constructions (e.g., subject-verb agreement) will boost the model's robustness on them in English, with a negligible loss of perplexity. The key to our success is an additional margin loss between the log-likelihoods of a correct word and an incorrect word. We then provide a detailed analysis of the trained models. One of our findings is the difficulty of object-relative clauses for RNNs. We find that even with our direct learning signals the models still suffer from resolving agreement across an object-relative clause. Augmentation of training sentences involving the constructions somewhat helps, but the accuracy still does not reach the level of subjectrelative clauses. Although not directly cognitively appealing, our method can be a tool to analyze the true architectural limitation of neural models on challenging linguistic constructions.

Cross-Linguistic Syntactic Evaluation of Word Prediction Models

“…This suggests that all monolingual models learned the basic facts of agreement, and were able to apply them to the vocabulary items in our materials. At the other end of the spectrum, performance was only slightly higher than chance in the Across an Object Relative Clause condition for all languages except German, suggesting that LSTMs tend to struggle with center embedding-that is, when a subject-verb dependency is nested within another dependency of the same kind (Marvin and Linzen, 2018;Noji and Takamura, 2020).…”

Understanding by Understanding Not: Modeling Negation in Language Models

“…Our work is in this spirit for negations. Noji and Takamura (2020) propose taking advantage of negative examples and unlikelihood in the training of language models to increase their syntactic abilities. Similarly, Min et al (2020) show the effectiveness of syntactic data augmentation in the case of robustness in NLI.…”

Learning with Contrastive Examples for Data-to-Text Generation

“…where δ is the margin between the log-likelihoods of x and x * . This was originally proposed for analyzing the syntactic abilities of language models (Noji and Takamura, 2020). This loss is useful for developing better language models.…”

“…Huang et al (2018) introduced a margin loss that estimates the quality of each beam-searched candidate by comparing it with the reference sentence. More recently, Noji and Takamura (2020) showed negative examples help to improve the syntactic ability of neural language models. They created negative instances from original instances by injecting a grammatical error and used them to calculate a margin loss that will be added to the cross-entropy loss.…”

“…Huang et al (2018) introduced a margin loss to penalize sentences in a beam, assuming that the generated sentences are imperfect. Noji and Takamura (2020) used synthesized ungrammatical sentences in addition to the originals to improve the syntactic ability of language models.…”