Review Article: On Vocabulary Richness

Wimmer, Gejza; Altmann, Gabriel

doi:10.1076/jqul.6.1.1.4148

Cited by 24 publications

(16 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The importance of looking at the fluctuations around Heaps' law is that this law is used in different applications [21], e.g., (i) to optimize the memory allocation in inverse indexing algorithms [22]; (ii) to estimate the vocabulary of a language [23,24]; and (iii) to compare the vocabulary richness of documents with different lengths [25][26][27]. Beyond linguistic applications, scalings of the number of unique items as a function of database size similar to Heaps' law have been observed in other domains, e.g.…”

Section: Introductionmentioning

confidence: 99%

Scaling laws and fluctuations in the statistics of word frequencies

Gerlach

Altmann

2014

New J. Phys.

View full text Add to dashboard Cite

In this paper we combine statistical analysis of written texts and simple stochastic models to explain the appearance of scaling laws in the statistics of word frequencies. The average vocabulary of an ensemble of fixed-length texts is known to scale sub-linearly with the total number of words (Heaps' law). Analyzing the fluctuations around this average in three large databases (Google-ngram, English Wikipedia, and a collection of scientific articles) we find that the standard deviation scales linearly with the average (Taylor's law), in contrast to the prediction of decaying fluctuations obtained using simple sampling arguments. We explain both scaling laws (Heaps and Taylor) by modeling the usage of words by a Poisson process with a fattailed distribution of word-frequencies (Zipf's law) and topic-dependent frequencies of individual words (as in topic models). Taking into account topical variations lead to quenched averages, turn the vocabulary size a non-self-averaging quantity, and explain the empirical observations. For the numerous practical applications relying on estimations of vocabulary size, our results show that uncertainties remain large even for long texts. We show how to account for these uncertainties in measurements of lexical richness of texts with different lengths.

show abstract

Section: Introductionmentioning

confidence: 99%

Scaling laws and fluctuations in the statistics of word frequencies

Gerlach

Altmann

2014

New J. Phys.

View full text Add to dashboard Cite

show abstract

“…VOCABULARY RICHNESS MEASURES Panas (2007) pointed out that "the index of lexical richness is a fundamental parameter describing the word frequency structure of a text", and noted that Wimmer and Altmann (1999), in a review article on vocabulary richness, discussed four different approaches to study the indices of vocabulary richness:…”

Section: Word Frequency Distributionmentioning

confidence: 99%

Mathematical Modelling of the Pattern of Occurrence of Words in Different Corpora of the Hindi Language^∗

Pande

Dhami

2013

Journal of Quantitative Linguistics

View full text Add to dashboard Cite

The incidence of different components of language in natural language texts are not arbitrarily organized but tend to obey particular laws which enable us to explain characteristic features of human language. The present paper is an attempt to analyse and model the pattern of occurrence of words in the Hindi language. Various kinds of corpora have been selected from different sources for the study, and the occurrence of words in these corpora has been observed for a variety of properties such as: frequencies, vocabulary measures, and pattern of initials of words relative to the subsequent matra.

show abstract

“…Furthermore, the calculation offers a solution to the much-debated question of vocabulary richness (Hubert & Labbé, 1994;Labbé, 1998;Wimmer & Altmann, 1999 however, when speaking or writing about things which do not really matter to him, the diversity of the author's vocabulary decreases significantly. Genre must also be taken into account: a person does not talk in private conversation the same way in which he writes for a scientific journal.…”

Section: Interpretation Of the Chartmentioning

confidence: 99%

“…Vocabulary growth is a well known topic in quantitative linguistics (Wimmer & Altmann, 1999). In any natural text, the rate at which new types appear is very high at the beginning and decreases slowly, while remaining positive even in extremely long works (Hubert-Labbé, 1988b, Hubert-Labbé, 1994.…”

Section: Vocabulary Growthmentioning

confidence: 99%

Automatic Segmentation of Texts and Corpora

Labbé

Hubert

2004

Journal of Quantitative Linguistics

View full text Add to dashboard Cite

Segmentation of large textual corpora is one of the major questions asked of literary studies. We present a combination of two relevant methods. First, vocabulary growth analysis highlights the main discontinuities in a work. Second, these results are supplemented with the analysis of variations in vocabulary diversity within corpora. A segmentation algorithm, associated with a test of validity, indicates the optimal succession in distinct stages. This method is applied to Racine's works and those of various other works in French. RésuméLe découpage des grands corpus de textes est l'une des questions cruciales posées aux études littéraires. Il est proposé une double méthode. L'analyse de la croissance du vocabulaire (typetoken ratio) met en lumière les principaux changements de rythme. Ces résultats sont complétés par l'étude de la diversité du vocabulaire. Un algorithme de segmentation, associé à un test de validité, indique le découpage optimal. La méthode est appliquée aux oeuvres de Racine, Corneille et aux discours du Général de Gaulle.

show abstract

Review Article: On Vocabulary Richness

Cited by 24 publications

References 0 publications

Scaling laws and fluctuations in the statistics of word frequencies

Scaling laws and fluctuations in the statistics of word frequencies

Mathematical Modelling of the Pattern of Occurrence of Words in Different Corpora of the Hindi Language^∗

Automatic Segmentation of Texts and Corpora

Contact Info

Product

Resources

About

Review Article: On Vocabulary Richness

Cited by 24 publications

References 0 publications

Scaling laws and fluctuations in the statistics of word frequencies

Scaling laws and fluctuations in the statistics of word frequencies

Mathematical Modelling of the Pattern of Occurrence of Words in Different Corpora of the Hindi Language∗

Automatic Segmentation of Texts and Corpora

Contact Info

Product

Resources

About

Mathematical Modelling of the Pattern of Occurrence of Words in Different Corpora of the Hindi Language^∗