Bias in Zipf's Law Estimators

Pilgrim, Charlie; Hills, Thomas T.

doi:10.48550/arxiv.2008.00903

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This estimator has the benefit of being widely used and well known. It shows bias (as do all Zipf estimators) [53], but the bias is systematic so is not critical for trend analysis.…”

Section: Draftmentioning

confidence: 99%

Information Foraging in the Attention Economy Drives the Rising Entropy of English

Pilgrim¹,

Guo²,

Hills³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Over the past 200 years, rising rates of information proliferation have created new environments for information competition and, consequently, new selective forces on information evolution. These forces influence the information diet available to consumers, who in turn choose what to consume, creating a feedback process similar to that seen in many ecosystems. As a first step towards understanding this relationship, we apply animal foraging models of diet choice to describe the evolution of long and short form media in response to human preferences for maximising utility rate. The model describes an increase in information rate (i.e., entropy) in response to information proliferation, as well as differences in entropy between short-form and long-form media (such as social media and books, respectively). We find evidence for a steady increase in word entropy in diverse media categories since 1900, as well as an accelerated entropy increase in short-form media. Overall the evidence suggests an increasingly competitive battle for our attention that is having a lasting influence on the evolution of language and communication systems.

show abstract

“…This estimator has the benefit of being widely used and well known. It shows bias (as do all Zipf estimators) [53], but the bias is systematic so is not critical for trend analysis.…”

Section: Draftmentioning

confidence: 99%