Generalization from correlated sets of patterns in the perceptron

Abstract

Generalization is a central aspect of learning theory. Here, we propose a framework that explores an auxiliary task-dependent notion of generalization, and attempts to quantitatively answer the following question: given two sets of patterns with a given degree of dissimilarity, how easily will a network be able to "unify" their interpretation? This is quantified by the volume of the configurations of synaptic weights that classify the two sets in a similar manner. To show the applicability of our idea in a concrete setting, we compute this quantity for the perceptron, a simple binary classifier, using the classical statistical physics approach in the replica-symmetric ansatz. In this case, we show how an analytical expression measures the "distance-based capacity", the maximum load of patterns sustainable by the network, at fixed dissimilarity between patterns and fixed allowed number of errors. This curve indicates that generalization is possible at any distance, but with decreasing capacity. We propose that a distance-based definition of generalization may be useful in numerical experiments with real-world neural networks, and to explore computationally sub-dominant sets of synaptic solutions. arXiv:1903.06818v1 [cond-mat.dis-nn]

Solvable Model for the Linear Separability of Structured Data

“…For , the recurrence Equation ( 51 ) becomes By substituting given by Equation ( 53 ) and solving the recurrence we obtain, after some algebra, where is the second structure parameter that is defined in Equation ( 18 ). Finally, by combining the leading order expansions ( 50 ) and the moments ( 53 ) and ( 55 ), and by rescaling, as in Equation ( 31 ), we have the following explicit expressions for the two main metrics of separability as functions of the multiplicity k and the structure parameters and : For data that are structured as pairs of points, , Equation ( 56 ) gives the storage capacity of an ensemble of segments; this special result was first obtained, by means of replica calculations, in [ 44 ], and it was then rediscovered in other contexts in [ 8 , 45 ].…”

Critical properties of the SAT/UNSAT transitions in the classification problem of structured data

“…2 and 3 report the numerical solutions of these saddle-point equations for the volumes, respectively, (2) and (3). The limiting values of α at the saddle point for q M → 1 can be worked out analytically (see [12,33]); they are, respectively,…”

Statistical learning theory of structured data

“…Integrating data structure within the framework of statistical mechanics is relatively straightforward and usually follows two steps: (i) define a generative model for the data, given in terms of a nonfactorized joint probability distribution P (X p , Y p ); (ii) compute averages over the measure P (the "disorder"); this is what was done for instance in [5,12,42]. How to best address data dependence in the SLT formalism, instead, is a debated issue.…”