On the State of the Art of Evaluation in Neural Language Models

Abstract

Ongoing innovations in recurrent neural network architectures have provided a steady influx of apparently state-of-the-art results on language modelling benchmarks. However, these have been evaluated using differing codebases and limited computational resources, which represent uncontrolled sources of experimental variation. We reevaluate several popular architectures and regularisation methods with large-scale automatic black-box hyperparameter tuning and arrive at the somewhat surprising conclusion that standard LSTM architectures, when properly regularised, outperform more recent models. We establish a new state of the art on the Penn Treebank and Wikitext-2 corpora, as well as strong baselines on the Hutter Prize dataset.

Evaluating deep transfer learning for whole-brain cognitive decoding

“…While this finding might seem trivial at first, it has far-reaching consequences, as it brings forward questions about the reproducibility of inferences about the mapping between brain activity and cognitive states that are drawn from the interpretation of cognitive decoding decisions of DL models. Recent empirical work in DL research has demonstrated that the convergence of DL models, and thereby the specifics of their learned mapping between input data and target signal, is dependent on many non-deterministic aspects of the training process, such as random seeds and random weight initializations (Dodge et al, 2019, Henderson et al, 2018, Lucic et al, 2018, Reimers and Gurevych, 2017 as well as the specific choices for other hyper-parameters, such as individual layer specifications and optimization algorithms Lucic et al (2018), Melis et al (2017), Zoph and Le (2017). It is thus possible that the mapping between cognitive states and brain activity that a DL model learns can change with these factors of the training.…”

Context-aware legal citation recommendation using deep learning

“…LSTMs [15] are a popular form of recurrent neural networks and serve as a well-known baseline for deep neural network models. Variants using LSTM remain competitive in various NLP tasks [22,25,29]. BiLSTM (Bi-directional LSTM) improves on the original LSTM by reading inputs in both forward and backward directions.…”

Using AntiPatterns to avoid MLOps Mistakes

“…For example, in [14], the authors question claimed advances in reinforcement learning research due to the lack of significance metrics and variability of results. In [24], the authors state that many years of claimed superiority in empirical performance in the field of language modeling is actually faulty and showcase that the well-known stacked LSTM architecture (with appropriate hyperparameter tuning) outperforms other more recent and more sophisticated architectures. In [26], the authors highlight a flaw in many previous research works (in the context of Bayesian deep learning) wherein a well established baseline (Monte Carlo dropout) when run to completion (i.e., when learning is not cut-off preemptively by setting it to terminate after a specified number of iterations), achieves similar or superior results compared to the very same models which showcased superior results when introduced.…”