Distinct flavors of Zipf's law and its maximum likelihood fitting: Rank-size and size-distribution representations

Abstract

In recent years, researchers have realized the difficulties of fitting power-law distributions properly. These difficulties are higher in Zipfian systems, due to the discreteness of the variables and to the existence of two representations for these systems, i.e., two versions depending on the random variable to fit: rank or size. The discreteness implies that a power law in one of the representations is not a power law in the other, and vice versa. We generate synthetic power laws in both representations and apply a state-of-the-art fitting method to each of the two random variables. The method (based on maximum likelihood plus a goodness-of-fit test) does not fit the whole distribution but the tail, understood as the part of a distribution above a cutoff that separates non-power-law behavior from power-law behavior. We find that, no matter which random variable is power-law distributed, using the rank as the random variable is problematic for fitting, in general (although it may work in some limit cases). One of the difficulties comes from recovering the "hidden" true ranks from the empirical ranks. On the contrary, the representation in terms of the distribution of sizes allows one to recover the true exponent (with some small bias when the underlying size distribution is a power law only asymptotically).

Lognormals, Power Laws and Double Power Laws in the Distribution of Frequencies of Harmonic Codewords from Classical Music

“…The random variable to fit is the absolute frequency n of the codewords (types); this choice is not obvious in Zipfian systems, see Ref. [30]. The fitting method is the one in Refs.…”

“…Although we could have fitted the power laws in the discrete case [13,30], we have considered the continuous case instead, in order to compare on equal footing with the (truncated) lognormal distribution defined below, which is continuous. For high enough values of n, the distinction between continuous and discrete random variables becomes irrelevant, but not for small values of n.…”

“…An annoying side effect of the lack of proper statistical tools is the recurrent debate about if the lognormal distribution is superior or not to the power law to describe (some) Zipfian systems [27,28]. A further concern is some ambiguity in the definition of Zipf's law [13,29,30], which admits several mathematical formulations not strictly equivalent between them.…”

The global ocean size-spectrum from bacteria to whales

“…If we rank the sizes of individuals in order, then there are n individuals with frequency greater than or equal to that of the n th largest size. The rank and cumulative distribution are thus proportional, and since the axes have been reversed then the exponents of the distributions are inverse, which in the particular case of a slope of −1 does not matter (see also ( 125 ).…”

Heaps’ law and vocabulary richness in the history of classical music harmony

“…A quantity of relevance in this context is the entropy of the distribution of type frequency [44]. Each codeword type r (with r = 1, 2, .…”