Criticality of mostly informative samples: a Bayesian model selection approach

Haimovici, Ariel; Marsili, Matteo

doi:10.1088/1742-5468/2015/10/p10013

Cited by 29 publications

(86 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This relation suggests that samples with a higher value of the relevance permit to estimate a larger number of parameters, as also advocated in Ref. [2].…”

Section: Relation With Parametric Modelssupporting

confidence: 61%

Statistical criticality arises in most informative representations

Cubero

Marsili

et al. 2019

J. Stat. Mech.

Self Cite

View full text Add to dashboard Cite

We show that statistical criticality, i.e. the occurrence of power law frequency distributions, arises in samples that are maximally informative about the underlying generating process. In order to reach this conclusion, we first identify the frequency with which different outcomes occur in a sample, as the variable carrying useful information on the generative process. The entropy of the frequency, that we call relevance, provides an upper bound to the number of informative bits. This differs from the entropy of the data, that we take as a measure of resolution. Samples that maximise relevance at a given resolution -that we call maximally informative samples -exhibit statistical criticality. In particular, Zipf's law arises at the optimal trade-off between resolution (i.e. compression) and relevance. As a byproduct, we derive a bound of the maximal number of parameters that can be estimated from a dataset, in the absence of prior knowledge on the generative model. Furthermore, we relate criticality to the statistical properties of the representation of the data generating process. We show that, as a consequence of the concentration property of the Asymptotic Equipartition Property, representations that are maximally informative about the data generating process are characterised by an exponential distribution of energy levels. This arises from a principle of minimal entropy, that is conjugate of the maximum entropy principle in statistical mechanics. This explains why statistical criticality requires no parameter fine tuning in maximally informative samples.When data are generated as independent draws from a parametric distribution, one can draw a sharp distinction between noise and useful information, that part of the data that can be used to estimate the generative model. Useful information is concentrated in sufficient statistics, which are those variables whose empirical value suffices to fully estimate the model's parameter [1]. The first aim of this paper is to draw the same distinction in the case where the model is not known. In this case, we show that the information on the 1 arXiv:1808.00249v5 [physics.data-an]

show abstract

“…This relation suggests that samples with a higher value of the relevance permit to estimate a larger number of parameters, as also advocated in Ref. [2].…”

Section: Relation With Parametric Modelssupporting

confidence: 61%

Statistical criticality arises in most informative representations

Cubero

Marsili

et al. 2019

J. Stat. Mech.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Broad distributions of spike frequencies, characterised by a high MSR, exhibit a stochastic variablility that requires richer parametric models, as shown in Ref. [19]. In a decoding perspective, these non-trivial distributions afford a higher degree of distinguishability of neural responses to a given stimuli or behaviour.…”

Section: Discussionmentioning

confidence: 99%

“…In order to quantify this information, we used the ideas in Refs. [19] and [32] to hypothesise that neurons having such non-trivial temporal structures, as manifested by broad distributions of the neural firing behaviour, are important to the representations that the brain region encodes. At a given resolution, as defined in Eq.…”

Section: Discussionmentioning

confidence: 99%

“…This intuition, discussed theoretically in Refs. [19] and [32], has also been used to identify biologically and evolutionary relevant amino acid sites in protein sequences [35]. Indeed, in spite of all advances in sequencing techniques, the genomes from which we can learn are only those left to us by evolution.…”

Section: Discussionmentioning

confidence: 99%

“…Ref. [19] [20], that we take as a measure of the relevance of neuron i for the freely-behaving animals being studied, at time scale ∆t.…”

Section: Multi-scale Relevancementioning

confidence: 99%

See 2 more Smart Citations

Finding informative neurons in the brain using Multi-Scale Relevance

Cubero

Marsili

Roudi

2018

Preprint

Self Cite

View full text Add to dashboard Cite

We propose a metric -called Multi-Scale Relevance (MSR) -to score neurons for their prominence in encoding for the animal's behaviour that is being observed in a multi-electrode array recording experiment. The MSR assumes that relevant neurons exhibit a wide variability in their dynamical state, in response to the external stimulus, across different time scales. It is a non-parametric, fully featureless indicator, in that it uses only the time stamps of the firing activity, without resorting to any a priori covariate or invoking any specific tuning curve for neural activity. We test the method on data from freely moving rodents, where we found that neurons having low MSR tend to have low mutual information and low firing sparsity across the correlates that are believed to be encoded by the region of the brain where the recordings were made. In addition, neurons with high MSR contain significant information on spatial navigation and allow to decode spatial position or head direction as efficiently as those neurons whose firing activity has high mutual information with the covariate to be decoded.

show abstract

Multiscale relevance and informative encoding in neuronal spike trains

2020

Self Cite

View full text Add to dashboard Cite

Neuronal responses to complex stimuli and tasks can encompass a wide range of time scales. Understanding these responses requires measures that characterize how the information on these response patterns are represented across multiple temporal resolutions. In this paper we propose a metric -which we call multiscale relevance (MSR) -to capture the dynamical variability of the activity of single neurons across different time scales. The MSR is a non-parametric, fully featureless indicator in that it uses only the time stamps of the firing activity without resorting to any a priori covariate or invoking any specific structure in the tuning curve for neural activity. When applied to neural data from the mEC and from the ADn and PoS regions of freely-behaving rodents, we found that neurons having low MSR tend to have low mutual information and low firing sparsity across the correlates that are believed to be encoded by the region of the brain where the recordings were made. In addition, neurons with high MSR contain significant information on spatial navigation and allow to decode spatial position or head direction as efficiently as those neurons whose firing activity has high mutual information with the covariate to be decoded and significantly better than the set of neurons with high local variations in their interspike intervals. Given these results, we propose that the MSR can be used as a measure to rank and select neurons for their information content without the need to appeal to any a priori covariate.

show abstract

Criticality of mostly informative samples: a Bayesian model selection approach

Cited by 29 publications

References 18 publications

Statistical criticality arises in most informative representations

Statistical criticality arises in most informative representations

Finding informative neurons in the brain using Multi-Scale Relevance

Multiscale relevance and informative encoding in neuronal spike trains

Contact Info

Product

Resources

About