Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data

Abstract

In many applications, one works with neural network models trained by someone else. For such pretrained models, one may not have access to training data or test data. Moreover, one may not know details about the model, e.g., the specifics of the training data, the loss function, the hyperparameter values, etc. Given one or many pretrained models, it is a challenge to say anything about the expected performance or quality of the models. Here, we address this challenge by providing a detailed meta-analysis of hundreds of publicly available pretrained models. We examine norm-based capacity control metrics as well as power law based metrics from the recently-developed Theory of Heavy-Tailed Self Regularization. We find that norm based metrics correlate well with reported test accuracies for well-trained models, but that they often cannot distinguish well-trained versus poorly trained models. We also find that power law based metrics can do much better—quantitatively better at discriminating among series of well-trained models with a given architecture; and qualitatively better at discriminating well-trained versus poorly trained models. These methods can be used to identify when a pretrained neural network has problems that cannot be detected simply by examining training/test accuracies.

Post-mortem on a deep learning contest: a Simpson's paradox and the complementary roles of scale metrics versus shape metrics

“…Broader picture. Overall, our analysis explains the success of previously-introduced metrics that combine norm information and shape/correlational information [6,7] as well as previouslyobserved peculiarities of norm-based metrics [15]. Our results also highlight the need to go beyond one-size-fits-all metrics based on upper bounds from generalization theory to describe the performance of SOTA NNs, as well as the evaluation of models by expensive retraining on test data.…”

“…1 The Contest [2,3] made available a data set consisting of pre-trained NN models. This Contest data was more narrow than another corpus analyzed recently [6,7], which used Heavy-Tailed Self-Regularization (HT-SR) theory [8,9,10] to analyze hundreds of state-of-the-art (SOTA) models from CV and natural language processing (NLP). However, the Contest data was more detailed, in the sense there were lower quality models, models in which hyperparameters were suboptimally chosen, etc.…”

“…The precise functional form of the tail is less relevant for us, but for simplicity we will consider parameters from truncated power laws that have which is based on statistical mechanics 5 and heavy-tailed random matrix theory. They do extremely well at predicting trends in the quality of SOTA NNs, even without access to training or testing data [6,7].…”

“…Our results also highlight the need to go beyond one-size-fits-all metrics based on upper bounds from generalization theory to describe the performance of SOTA NNs, as well as the evaluation of models by expensive retraining on test data. Test data can be extremely expensive to acquire, and there is a need to evaluate models without (much or any) test data [6,7]. We aim to develop a theory of generalization that practioners can use in their day-to-day training, fine-tuning, validation, and evaluation of models.…”

Automatic Assessment of Emotion Dysregulation in American, French, and Tunisian Adults and New Developments in Deep Multimodal Fusion: Cross-sectional Study

“…Furthermore, we question whether a hold-out sample, proportional in size to our overall sample, would have been a better unbiased estimator (how can a sample with a size of around 30 be taken as representative of the whole population?). In the future, we will look to the works of Martin and Corneanu [40,41] that unlock estimating generalization performance directly from the characteristics of the model itself. We are already working on a criterion inspired by their work, which we call the network engagement criterion.…”