Solvable Model for the Linear Separability of Structured Data

Abstract

Linear separability, a core concept in supervised machine learning, refers to whether the labels of a data set can be captured by the simplest possible machine: a linear classifier. In order to quantify linear separability beyond this single bit of information, one needs models of data structure parameterized by interpretable quantities, and tractable analytically. Here, I address one class of models with these properties, and show how a combinatorial method allows for the computation, in a mean field approximation, of two useful descriptors of linear separability, one of which is closely related to the popular concept of storage capacity. I motivate the need for multiple metrics by quantifying linear separability in a simple synthetic data set with controlled correlations between the points and their labels, as well as in the benchmark data set MNIST, where the capacity alone paints an incomplete picture. The analytical results indicate a high degree of “universality”, or robustness with respect to the microscopic parameters controlling data structure.

Inversion dynamics of class manifolds in deep learning reveals tradeoffs underlying generalisation

“…Intuitively, separation of the two classes by the last layer is facilitated whenever M + (T ) and M − (T ), at the final epoch T , are small or far apart. This intuition is confirmed by analytical results obtained for the perceptron [13,29]. Our analysis is based on a simple descriptor of manifold extension, the gyration radius, a metric proxy of the set's extension in Euclidean space.…”

Universal mean-field upper bound for the generalization gap of deep neural networks

“…The goal of theorems in SLT is to provide distribution-independent uniform bounds on the deviation between the generalisation and training errors. The formulation and the derivation of these theorems reveal a source of possible reasons for their poor quantitative performance: (i) empirically relevant data distributions may lead to smaller typical deviations than the worst possible case [27][28][29][30][31]; (ii) uniform bounds hold for all possible functions in the model, but better bounds may hold when one restricts the analysis to functions that perform well on specific (and significative) training sets.…”

Critical properties of the SAT/UNSAT transitions in the classification problem of structured data

“…In other words, an admissible classification is such that it assigns the same label to sufficiently similar, or correlated, inputs. In [11][12][13], it is conjectured that a measure of the number of the admissible classifications alone within a certain hypothesis class should be a better bound for the generalization error than the classic data-independent measures of complexity. Indeed, while the number of all the realizable classifications usually grows monotonically with the number of inputs to classify (more points, more ways to label them), the admissibility constraint adds a competing effect of excluded volume (more points, less space available for a separating surface enforcing an admissible classification), which is eventually dominant as the number of inputs grows.…”