Similarity of Symbol Frequency Distributions with Heavy Tails

Abstract

Quantifying the similarity between symbolic sequences is a traditional problem in information theory which requires comparing the frequencies of symbols in different sequences. In numerous modern applications, ranging from DNA over music to texts, the distribution of symbol frequencies is characterized by heavy-tailed distributions (e.g., Zipf's law). The large number of low-frequency symbols in these distributions poses major difficulties to the estimation of the similarity between sequences; e.g., they hinder an accurate finite-size estimation of entropies. Here, we show analytically how the systematic (bias) and statistical (fluctuations) errors in these estimations depend on the sample size N and on the exponent. of the heavy-tailed distribution. Our results are valid for the Shannon entropy (alpha = 1), its corresponding similarity measures (e.g., the Jensen-Shanon divergence), and also for measures based on the generalized entropy of order a. For small alpha's, including alpha = 1, the errors decay slower than the 1/N decay observed in short-tailed distributions. For a larger than a critical value alpha* = 1 + 1/gamma <= 2, the 1/N decay is recovered. We show the practical significance of our results by quantifying the evolution of the English language over the last two centuries using a complete alpha spectrum of measures. We find that frequent words changemore slowly than less frequent words and that alpha = 2 provides the most robust measure to quantify language change

The Entropy of Words—Learnability and Expressivity across More than 1000 Languages

“…For example, several recent studies engage in establishing information-theoretic and corpus-based methods for linguistic typology, i.e., classifying and comparing languages according to their information encoding potential [10,[12][13][14][15][16], and how this potential evolves over time [17][18][19]. Similar methods have been applied to compare and distinguish non-linguistic sequences from written language [20,21], though it is controversial whether this helps with more fine-grained distinctions between symbolic systems and written language [22,23].…”

Statistical properties of links of network: A survey on the shipping lines of Worldwide Marine Transport Network

“…10. The Jensen-Shannon distance is a quantity to measure the similarity of different series, and is useful in complex system field [40,41]. For two networks with Laplacian matrix eigenvalue sequences P (x) and Q(x), respectively, which are serialized by Gaussian kernel g(x, γ) = 1 √ 2πσ 2 exp(− (x−γ) 2 2σ 2 ), the Jensen-Shannon distance can be expressed as:…”

The Entropy of Words&mdash;Learnability and Expressivity across More Than 1000 Languages

“…For example, several recent studies engage in establishing information-theoretic and corpus-based methods for linguistic typology, i.e. classifying and comparing languages according to their information encoding potential [10,[12][13][14][15][16], and how this potential evolves over time [17][18][19]. Similar methods have been applied to compare and distinguish non-linguistic sequences from written 2 of 34 language [20,21], though it is controversial whether this helps with more fine-grained distinctions between symbolic systems and written language [22,23].…”