Generalization in Generation: A closer look at Exposure Bias

Schmidt, Florian

doi:10.18653/v1/d19-5616

Cited by 45 publications

(27 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1. b) If the model only observes states resulting from correct past decisions at training time, it will not be prepared to recover from its own mistakes during prediction, suffering from exposure bias (Schmidt, 2019;Fried and Klein, 2018). In the experiment section, we demonstrate how this phenomenon will significantly hurt the language model performance and, to a lesser extent, also hurt the parsing performance.…”

Section: Syntactic Ordered Memorymentioning

confidence: 95%

Explicitly Modeling Syntax in Language Models with Incremental Parsing and a Dynamic Oracle

Shen¹,

Tan²,

Sordoni³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Syntax is fundamental to our thinking about language. Failing to capture the structure of input language could lead to generalization problems and over-parametrization. In the present work, we propose a new syntax-aware language model: Syntactic Ordered Memory (SOM). The model explicitly models the structure with an incremental parser and maintains the conditional probability setting of a standard language model (left-to-right). To train the incremental parser and avoid exposure bias, we also propose a novel dynamic oracle, so that SOM is more robust to wrong parsing decisions. Experiments show that SOM can achieve strong results in language modeling, incremental parsing and syntactic generalization tests, while using fewer parameters than other models.

show abstract

Section: Syntactic Ordered Memorymentioning

confidence: 95%

Explicitly Modeling Syntax in Language Models with Incremental Parsing and a Dynamic Oracle

Shen¹,

Tan²,

Sordoni³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

show abstract

“…The predictions for MP, ME and Qual spans in Stage 3 are heavily dependent on the Q spans from Stage 1, and there does not exist any mechanism to rectify errors in Stage 1 later, in our approach. There is also an exposure bias (Schmidt, 2019;Galloway et al, 2019) as the model is trained on the ground truth, while tested on the predicted Q spans. Moreover, we believe that having common weights between the BERT models of Stage 1 and Stage 3 will not only make our approach faster and lighter, but also more performant through multi-task learning.…”

Section: Future Workmentioning

confidence: 99%

KGP at SemEval-2021 Task 8: Leveraging Multi-Staged Language Models for Extracting Measurements, their Attributes and Relations

Karia¹,

Kaushal²,

Mallick³

2021

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)

View full text Add to dashboard Cite

SemEval-2021 Task 8: MeasEval aims at improving the machine understanding of measurements in scientific texts through a set of entity and semantic relation extraction sub-tasks on identifying quantity spans along with various attributes and relationships. This paper describes our system, consisting of a three-stage pipeline, that leverages pre-trained language models to extract the quantity spans in the text, followed by intelligent templates to identify units and modifiers. Finally, it identifies the quantity attributes and their relations using language models boosted with a feature re-using hierarchical architecture and multi-task learning. Our submission significantly outperforms the baseline, with the best model from the postevaluation phase delivering more than 100% increase on F1 (Overall) from the baseline.

show abstract

“…It is important to mention that, all smoothing methods mentioned in this paper are only used during training, while during testing, the one-hot vectors are again used, following the idea that a model trained with smoothed representation of words should better generalize during testing. This can be further related to the topic of exposure bias (Schmidt, 2019), where the training-testing mismatch may lead to unseen or corrupted context during search. Although we do not conduct experiments to further validate the point, qualitatively we think input smoothing effectively lets the model see more unique contexts during training, mitigating the exposure bias problem.…”

Section: Input Smoothing + Output Smoothingmentioning

confidence: 99%

Unifying Input and Output Smoothing in Neural Machine Translation

Gao

Liao²,

Ney

2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

Soft contextualized data augmentation is a recent method that replaces one-hot representation of words with soft posterior distributions of an external language model, smoothing the input of neural machine translation systems. Label smoothing is another effective method that penalizes over-confident model outputs by discounting some probability mass from the true target word, smoothing the output of neural machine translation systems. Having the benefit of updating all word vectors in each optimization step and better regularizing the models, the two smoothing methods are shown to bring significant improvements in translation performance. In this work, we study how to best combine the methods and stack the improvements. Specifically, we vary the prior distributions to smooth with, the hyperparameters that control the smoothing strength, and the token selection procedures. We conduct extensive experiments on small datasets, evaluate the recipes on larger datasets, and examine the implications when back-translation is further used. Our results confirm cumulative improvements when input and output smoothing are used in combination, giving up to +1.9 BLEU scores on standard machine translation tasks and reveal reasons why these smoothing methods should be preferred.

show abstract

Generalization in Generation: A closer look at Exposure Bias

Cited by 45 publications

References 26 publications

Explicitly Modeling Syntax in Language Models with Incremental Parsing and a Dynamic Oracle

Explicitly Modeling Syntax in Language Models with Incremental Parsing and a Dynamic Oracle

KGP at SemEval-2021 Task 8: Leveraging Multi-Staged Language Models for Extracting Measurements, their Attributes and Relations

Unifying Input and Output Smoothing in Neural Machine Translation

Contact Info

Product

Resources

About