Counting the learnable functions of geometrically structured data

Rotondo, Pietro; Lagomarsino, Marco Cosentino

doi:10.1103/physrevresearch.2.023169

Cited by 21 publications

(30 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Fixing a model of data structure in this context means fixing a generative model of data. Here, I use the model first introduced in [ 28 ]. This should not be considered to be a realistic model of real data sets.…”

Section: Parameterized Model Of Structured Datamentioning

confidence: 99%

“…If

, then the quantity that enters the equations will be the mean of

over all the pairs in the multiplet. It can be shown that

, as a function of the overlaps

, does not explicitly depend on the dimensionality n [ 28 ]; this property greatly simplifies the analytical computations.…”

Section: Parameterized Model Of Structured Datamentioning

confidence: 99%

“…An important class of models assumes that data points are samples of probability distributions that are supported on extended object manifold, which represent all possible variations of an input that should have no effect on its classification (e.g., differences in brightness of a photo, differences in aspect ratio of a handwritten digit) [ 27 ]. Recently, a useful parameterization of object manifolds was introduced that is amenable to analytical computations [ 28 ]; it will be described in detail below. In a data science perspective, these approaches are motivated by the empirical observation that data sets usually lie on low-dimensional manifolds, whose “intrinsic dimension” is a measure of the number of latent degrees of freedom [ 29 , 30 , 31 ].…”

Section: Introductionmentioning

confidence: 99%

“…Most works concerned with data structure and object manifolds (in particular, Refs. [ 8 , 27 , 28 ]) focus on a single descriptor of linear separability, namely the storage capacity

. Informally, the storage capacity measures the maximum number of points that a classifier can reliably classify; in statistical mechanics, it signals the transition, in the thermodynamic limit, between the SAT and UNSAT phases of the random satisfiability problem related to the linear separability of random data [ 32 ].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Solvable Model for the Linear Separability of Structured Data

Gherardi

2021

Entropy

Self Cite

View full text Add to dashboard Cite

Linear separability, a core concept in supervised machine learning, refers to whether the labels of a data set can be captured by the simplest possible machine: a linear classifier. In order to quantify linear separability beyond this single bit of information, one needs models of data structure parameterized by interpretable quantities, and tractable analytically. Here, I address one class of models with these properties, and show how a combinatorial method allows for the computation, in a mean field approximation, of two useful descriptors of linear separability, one of which is closely related to the popular concept of storage capacity. I motivate the need for multiple metrics by quantifying linear separability in a simple synthetic data set with controlled correlations between the points and their labels, as well as in the benchmark data set MNIST, where the capacity alone paints an incomplete picture. The analytical results indicate a high degree of “universality”, or robustness with respect to the microscopic parameters controlling data structure.

show abstract

Section: Parameterized Model Of Structured Datamentioning

confidence: 99%

“…If

, then the quantity that enters the equations will be the mean of

over all the pairs in the multiplet. It can be shown that

, as a function of the overlaps

, does not explicitly depend on the dimensionality n [ 28 ]; this property greatly simplifies the analytical computations.…”

Section: Parameterized Model Of Structured Datamentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

“…Most works concerned with data structure and object manifolds (in particular, Refs. [ 8 , 27 , 28 ]) focus on a single descriptor of linear separability, namely the storage capacity

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Solvable Model for the Linear Separability of Structured Data

Gherardi

2021

Entropy

Self Cite

View full text Add to dashboard Cite

show abstract

“…Analyzing training and generalization performance in feed-forward and recurrent networks as a function of statistical and geometrical structure of a task remains an open problem both in computational neuroscience and statistical learning theory [28][29][30][31][32]. This calls for statistical models of the low-dimensional structure of data that are at the same time expressive and amenable to mathematical analyses.…”

Section: Introductionmentioning

confidence: 99%

Optimal learning with excitatory and inhibitory synapses

Ingrosso

2020

PLoS Comput Biol

View full text Add to dashboard Cite

Characterizing the relation between weight structure and input/output statistics is fundamental for understanding the computational capabilities of neural circuits. In this work, I study the problem of storing associations between analog signals in the presence of correlations, using methods from statistical mechanics. I characterize the typical learning performance in terms of the power spectrum of random input and output processes. I show that optimal synaptic weight configurations reach a capacity of 0.5 for any fraction of excitatory to inhibitory weights and have a peculiar synaptic distribution with a finite fraction of silent synapses. I further provide a link between typical learning performance and principal components analysis in single cases. These results may shed light on the synaptic profile of brain circuits, such as cerebellar structures, that are thought to engage in processing time-dependent signals and performing on-line prediction.

show abstract