Entropy and Inference, Revisited

Nemenman, Ilya; Shafee, Fariel; Bialek, William

doi:10.7551/mitpress/1120.003.0065

Cited by 78 publications

(39 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…as the Dirichlet distribution over the (K − 1) simplex with all parameters α 1 = • • • = α K = α. Sampling from this Dirichlet with a fixed α, however, has the undesirable effect of generating distributions with a very narrow distribution of entropy H(X) [59]. To generate distributions with a near-uniform of entropy, we sample α from a Nemenman-Shafee-Bialek (NSB) prior [60] p(α) ∝ Kψ(Kα + 1) + ψ(α + 1), for a distribution over an alphabet of size K. For simulations of n binary variables, we set K = 2 n , sample α using the equation above, and then sample p X from a symmetric Dirichlet using standard algorithms.…”

Section: Discussionmentioning

confidence: 99%

An operational information decomposition via synergistic disclosure

Rosas

Mediano²,

Rassouli³

et al. 2020

J. Phys. A: Math. Theor.

View full text Add to dashboard Cite

Multivariate information decompositions hold promise to yield insight into complex systems, and stand out for their ability to identify synergistic phenomena. However, the adoption of these approaches has been hindered by there being multiple possible decompositions, and no precise guidance for preferring one over the others. At the heart of this disagreement lies the absence of a clear operational interpretation of what synergistic information is. Here we fill this gap by proposing a new information decomposition based on a novel operationalisation of informational synergy, which leverages recent developments in the literature of data privacy. Our decomposition is defined for any number of information sources, and its atoms can be calculated using elementary optimisation techniques. The decomposition provides a natural coarse-graining that scales gracefully with the system’s size, and is applicable in a wide range of scenarios of practical interest.

show abstract

Section: Discussionmentioning

confidence: 99%

An operational information decomposition via synergistic disclosure

Rosas

Mediano²,

Rassouli³

et al. 2020

J. Phys. A: Math. Theor.

View full text Add to dashboard Cite

show abstract

“…Fixedsize sliding windows are varied from 1 to 5. Word entropy calculations were made using the NSB estimator [24].…”

Section: (H Von Neumann (G)) = H Von Neumann G ∪(Uv) − H Von Neumann ...mentioning

confidence: 99%

“…The maximum likelihood entropy estimator is known to underestimate the true entropy in practical applications [23]. A range of more advanced entropy estimators have been proposed to overcome this limitation [23][24][25][26]. Here, we used the NSB estimator [24] to calculate word entropy.…”

mentioning

confidence: 99%

See 1 more Smart Citation

On the von Neumann entropy of language networks: Applications to cross-linguistic comparisons

Vera¹,

Fuentealba²,

López³

et al. 2021

EPL

View full text Add to dashboard Cite

Words are not isolated entities within a language. In this paper, we measure the number of choices transmitted in natural language by means of the von Neumann entropy of language networks. This quantity, introduced in Quantum Information accounts, provides a detailed characterization of network complexities. The simulations are based on a large parallel corpus of 362 languages across 55 linguistic families (focusing on the sub-sample of 85 languages from the Americas). With this, we constructed language networks as a simple way to describe word connectivity patterns for each language. We studied several aspects of the von Neumann entropy of language networks. First, we discovered large groups of languages with low average degree and high von Neumann entropy. The results suggested also that large von Neumann entropy is associated with word entropy (as a proxy for morphological complexity), and is inversely related to degree regularity. This means that there are pressures at play that keep a balance between word morphological complexity and patterns of connections between words. We suggested also a strong influence of functional words on low von Neumann entropy languages. Our approach is thus a simple network-based contribution to establish cross-linguistic language comparisons from textual data.

show abstract

“…Using several corpora and tackling some problems of word entropy estimation [18], provided a public database of entropy values for 1259 languages. Since all entropy estimators are strongly correlated, for our experiments we used the entropy values provided by the NSB estimator [20].…”

Section: Information-theoretic Entropymentioning

confidence: 99%

The community structure of word co-occurrence networks: Experiments with languages from the Americas

Vera¹,

Palma²

2021

EPL

View full text Add to dashboard Cite

We study a set of algorithms to discover the community structure of networks for languages from the Americas. Our experiments are based on a parallel corpus which allows us to represent each language as a co-occurrence network. Four methods to calculate network modularity, as a measure of the quality of community structure, were used. We studied several aspects of the community structure of co-occurrence networks. First, we were able to construct the map of modularity variations across languages from the Americas. With this, we separated large groups of languages into low-and high-modularity families. We suggested also a strong influence of functional words on low-modularity languages. Finally, we found a strong relationship between word entropy values and modularity. Our approach is thus a simple network-based contribution to face data scarcity of languages which are in danger of disappearing.

show abstract

Entropy and Inference, Revisited

Cited by 78 publications

References 12 publications

An operational information decomposition via synergistic disclosure

An operational information decomposition via synergistic disclosure

On the von Neumann entropy of language networks: Applications to cross-linguistic comparisons

The community structure of word co-occurrence networks: Experiments with languages from the Americas

Contact Info

Product

Resources

About