The Missing Margin: How Sample Corruption Affects Distance to the Boundary in ANNs

Abstract

Classification margins are commonly used to estimate the generalization ability of machine learning models. We present an empirical study of these margins in artificial neural networks. A global estimate of margin size is usually used in the literature. In this work, we point out seldom considered nuances regarding classification margins. Notably, we demonstrate that some types of training samples are modelled with consistently small margins while affecting generalization in different ways. By showing a link with the minimum distance to a different-target sample and the remoteness of samples from one another, we provide a plausible explanation for this observation. We support our findings with an analysis of fully-connected networks trained on noise-corrupted MNIST data, as well as convolutional networks trained on noise-corrupted CIFAR10 data.Recent studies of the relationship between margin and generalization typically measure the average margins over the training set or some sampling thereof. In this work, we ask whether the margins of individual samples tend to reflect this average behaviour. Specifically, we focus on ANN-based classification models and introduce controlled noise This work is a preprint of a published paper by the same name [1], which it subsumes. This preprint is an extended version: it contains additional empirical evidence (clean and input-corrupted CNN models