Dependence of exponents on text length versus finite-size scaling for word-frequency distributions

Corral, √Ålvaro; Font-Clos, Francesc

doi:10.1103/physreve.96.022318

Cited by 12 publications

(11 citation statements)

References 41 publications

(113 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the other side, the theory of scaling analysis, following the authors of [ 21 , 29 ], allows us to compare the shape of the conditional distributions

for different values of ℓ . This theory has revealed a very powerful tool in quantitative linguistics, allowing in previous research to show that the shape of the word-frequency distribution does not change as a text increases its length [ 30 , 31 ].…”

Section: Corpus and Statistical Methodsmentioning

confidence: 99%

The Brevity Law as a Scaling Law, and a Possible Origin of Zipf’s Law for Word Frequencies

Corral

Serra

2020

Entropy

Self Cite

View full text Add to dashboard Cite

An important body of quantitative linguistics is constituted by a series of statistical laws about language usage. Despite the importance of these linguistic laws, some of them are poorly formulated, and, more importantly, there is no unified framework that encompasses all them. This paper presents a new perspective to establish a connection between different statistical linguistic laws.Characterizing each word type by two random variables, length (in number of characters) and absolute frequency, we show that the corresponding bivariate joint probability distribution shows a rich and precise phenomenology, with the type-length and the type-frequency distributions as its two marginals, and the conditional distribution of frequency at fixed length providing a clear formulation for the brevity-frequency phenomenon. The type-length distribution turns out to be well fitted by a gamma distribution (much better than with the previously proposed lognormal), and the conditional frequency distributions at fixed length display power-law-decay behavior with a fixed exponent α 1.4 and a characteristic-frequency crossover that scales as an inverse power δ 2.8 of length, which implies the fulfilment of a scaling law analogous to those found in the thermodynamics of critical phenomena. As a by-product, we find a possible model-free explanation for the origin of Zipf's law, which should arise as a mixture of conditional frequency distributions governed by the crossover length-dependent frequency.

show abstract

“…On the other side, the theory of scaling analysis, following the authors of [ 21 , 29 ], allows us to compare the shape of the conditional distributions

Section: Corpus and Statistical Methodsmentioning

confidence: 99%

The Brevity Law as a Scaling Law, and a Possible Origin of Zipf’s Law for Word Frequencies

Corral

Serra

2020

Entropy

Self Cite

View full text Add to dashboard Cite

show abstract

“…We can use a variation of the logarithmic-coefficient-ofvariation test (in fact, its original linear form, essentially) to rule out that the distribution of cluster size has an exponential tail, as was claimed in other contexts [43] (and already criticized in Refs. [44,45]). If we compute the usual residual In all cases the first crossing below the fifth percentile takes place for n cv around 100, which corresponds to n PL in Table I.…”

Section: B Residual Logarithmic Coefficient Of Variationmentioning

confidence: 99%

Truncated lognormal distributions and scaling in the size of naturally defined population clusters

2020

Self Cite

View full text Add to dashboard Cite

Using population data of high spatial resolution for a region in the south of Europe, we define cities by aggregating individuals to form connected clusters. The resulting cluster-population distributions show a smooth decreasing behavior covering six orders of magnitude. We perform a detailed study of the distributions, using state-of-the-art statistical tools. By means of scaling analysis we rule out the existence of a power-law regime in the low-population range. The logarithmic-coefficient-of-variation test allows us to establish that the power-law tail for high population, characteristic of Zipf's law, has a rather limited range of applicability. Instead, lognormal fits describe the population distributions in a range covering from a few dozen individuals to more than 1 × 10 6 (which corresponds to the population of the largest cluster).

show abstract

“…Another source of variation in the value of the resulting system size L tot is that this arises as a sum of independent power-law distributed sizes n. As the exponent of the power law γ is smaller than 2, the law of large numbers does not apply and the sum is not scaling linearly with the number of terms (types) V tot . Instead, the sum is broadly distributed, as expected from the generalized central limit theorem [64][65][66]. Table IV provides the results obtained from other examples simulating Zipf's law for sizes [Eq.…”

Section: Simulation Of Zipf's Law With Size As the Random Variablementioning

confidence: 86%

Distinct flavors of Zipf's law and its maximum likelihood fitting: Rank-size and size-distribution representations

2020

Self Cite

View full text Add to dashboard Cite

In recent years, researchers have realized the difficulties of fitting power-law distributions properly. These difficulties are higher in Zipfian systems, due to the discreteness of the variables and to the existence of two representations for these systems, i.e., two versions depending on the random variable to fit: rank or size. The discreteness implies that a power law in one of the representations is not a power law in the other, and vice versa. We generate synthetic power laws in both representations and apply a state-of-the-art fitting method to each of the two random variables. The method (based on maximum likelihood plus a goodness-of-fit test) does not fit the whole distribution but the tail, understood as the part of a distribution above a cutoff that separates non-power-law behavior from power-law behavior. We find that, no matter which random variable is power-law distributed, using the rank as the random variable is problematic for fitting, in general (although it may work in some limit cases). One of the difficulties comes from recovering the "hidden" true ranks from the empirical ranks. On the contrary, the representation in terms of the distribution of sizes allows one to recover the true exponent (with some small bias when the underlying size distribution is a power law only asymptotically).

show abstract

Dependence of exponents on text length versus finite-size scaling for word-frequency distributions

Cited by 12 publications

References 41 publications

The Brevity Law as a Scaling Law, and a Possible Origin of Zipf’s Law for Word Frequencies

The Brevity Law as a Scaling Law, and a Possible Origin of Zipf’s Law for Word Frequencies

Truncated lognormal distributions and scaling in the size of naturally defined population clusters

Distinct flavors of Zipf's law and its maximum likelihood fitting: Rank-size and size-distribution representations

Contact Info

Product

Resources

About