Random texts exhibit Zipf's-law-like word frequency distribution

Li, Wentian

doi:10.1109/18.165464

Cited by 375 publications

(342 citation statements)

References 3 publications

Supporting

Mentioning

315

Contrasting

Unclassified

Order By: Relevance

“…It has been shown that this Zipf-Mandelbrot law is also obeyed by so many random processes [69,70] that one has been sometimes ruling out any interestingly special character for linguistic studies. Nevertheless, it has been argued that it is possible to discriminate between human writings [71] and stochastic versions of texts precisely by looking at statistical properties of words that fall where Eq.…”

Section: Zipf Methodsmentioning

confidence: 99%

Equilibrium and dynamic methods when comparing an English text and its Esperanto translation

Ausloos¹

2008

Physica A: Statistical Mechanics and its Applications

View full text Add to dashboard Cite

A comparison of two english texts from Lewis Carroll, one (Alice in wonderland), also translated into esperanto, the other (Through a looking glass) are discussed in order to observe whether natural and artificial languages significantly differ from each other. One dimensional time series like signals are constructed using only word frequencies (FTS) or word lengths (LTS). The data is studied through (i) a Zipf method for sorting out correlations in the FTS and (ii) a Grassberger-Procaccia (GP) technique based method for finding correlations in LTS. Features are compared : different power laws are observed with characteristic exponents for the ranking properties, and the phase space attractor dimensionality. The Zipf exponent can take values much less than unity (ca. 0.50 or 0.30) depending on how a sentence is defined. This non-universality is conjectured to be a measure of the author style. Moreover the attractor dimension r is a simple function of the so called phase space dimension n, i.e., r = n λ , with λ = 0.79. Such an exponent should also conjecture to be a measure of the author creativity. However, even though there are quantitative differences between the original english text and its esperanto translation, the qualitative differences are very minutes, indicating in this case a translation relatively well respecting, along our analysis lines, the content of the author writing.

show abstract

Section: Zipf Methodsmentioning

confidence: 99%

Equilibrium and dynamic methods when comparing an English text and its Esperanto translation

Ausloos¹

2008

Physica A: Statistical Mechanics and its Applications

View full text Add to dashboard Cite

show abstract

“…This argument was first introduced by Miller [19,20] to explain the power law distribution of words in texts. Here we study another simple way to extract a power-law, in this case from a fluctuating or time dependent Poisson process.…”

mentioning

confidence: 99%

Conditions for the emergence of scaling in the inter-event time of uncorrelated and seasonal systems

Hidalgo

2006

Physica A: Statistical Mechanics and its Applications

View full text Add to dashboard Cite

Interevent times have been studied across various disciplines in search for correlations. In this paper we show analytical and numerical evidence that at the population level a power-law can be obtained by assuming poissonian agents with different characteristic times, and at the individual level by assuming poissonian agents that change the rates at which they perform an event in a random or deterministic fashion. The range in which we expect to see this behavior and the possible deviations from it are studied by considering the shape of the rate distribution.

show abstract

“…By employing linear regression, we estimate approximate value of s to be 1.053, which follows the empirical observations that s → 1 and H ∝ (|Q| + |T |) when applied to human language [11]. We derive a final, concise relationship:…”

Section: Token-based Canopiesmentioning

confidence: 99%

Online Social Network Profile Linkage

Zhang

Kan

Liu

et al. 2014

Information Retrieval Technology

View full text Add to dashboard Cite

Abstract. Piecing together social signals from people in different online social networks is key for downstream analytics. However, users may have different usernames in different social networks, making the linkage task difficult. To enable this, we explore a probabilistic approach that uses a domain-specific prior knowledge to address this problem of online social network user profile linkage. At scale, linkage approaches that are based on a naïve pairwise comparisons that have quadratic complexity become prohibitively expensive. Our proposed threshold-based canopying framework -named OPL -reduces this pairwise comparisons, and guarantees a upper bound theoretic linear complexity with respect to the dataset size. We evaluate our approaches on real-world, large-scale datasets obtained from Twitter and Linkedin. Our probabilistic classifier integrating prior knowledge into Naïve Bayes performs at over 85% F1-measure for pairwise linkage, comparable to state-of-the-art approaches.

show abstract

Random texts exhibit Zipf's-law-like word frequency distribution

Cited by 375 publications

References 3 publications

Equilibrium and dynamic methods when comparing an English text and its Esperanto translation

Equilibrium and dynamic methods when comparing an English text and its Esperanto translation

Conditions for the emergence of scaling in the inter-event time of uncorrelated and seasonal systems

Online Social Network Profile Linkage

Contact Info

Product

Resources

About