Critical properties of the SAT/UNSAT transitions in the classification problem of structured data

Abstract

The classification problem of structured data can be solved with different strategies: a supervised learning approach, starting from a labeled training set, and an unsupervised learning one, where only the structure of the patterns in the dataset is used to find a classification compatible with it. The two strategies can be interpreted as extreme cases of a semi-supervised approach to learn multi-view data, relevant for applications. In this paper I study the critical properties of the two storage problems associated with these tasks, in the case of the linear binary classification of doublets of points sharing the same label, within replica theory. While the first approach presents an SAT/UNSAT transition in a (marginally) stable replica-symmetric phase, in the second one the satisfiability line lies in a full replica-symmetry-broken phase. A similar behavior in the problem of learning with a margin is also pointed out.

Satisfiability transition in asymmetric neural networks

“…This deviates from the standard theory of random graphical models that assumes factors are independent random variables. Establishing the correct connection with the cavity method should shed light on the one assumption from GGY's calculation we have not explored in this paper, the mean-field like assumption (45). The very strong resemblance between the OPN and the MPN could mean that this assumption is too strong.…”

“…In that case, indeed, the transition always takes place for q 0 → 1: whenever the number of solutions of the CSP decreases, different replicas of the system become more and more correlated and their overlap tends to 1, at the point where a single solution is left. This picture remains unchanged even in CSPs where the RS ansatz is not correct (see for example [42][43][44][45]), where the RS q 0 → 1 line gives an upper bound on the true α c where the transition occurs.…”

“…We calculate volume of solutions for the one pattern model in the limit C → ∞ after N → ∞. We do the calculation directly with replicas without doing the mean-field assumption (45) of GGY used in [23]. We start from the fractional synaptic volume…”

Universal mean-field upper bound for the generalization gap of deep neural networks

“…The goal of theorems in SLT is to provide distribution-independent uniform bounds on the deviation between the generalisation and training errors. The formulation and the derivation of these theorems reveal a source of possible reasons for their poor quantitative performance: (i) empirically relevant data distributions may lead to smaller typical deviations than the worst possible case [27][28][29][30][31]; (ii) uniform bounds hold for all possible functions in the model, but better bounds may hold when one restricts the analysis to functions that perform well on specific (and significative) training sets.…”