vocd: A theoretical and empirical evaluation

Abstract

A reliable index of lexical diversity (LD) has remained stubbornly elusive for over 60 years. Meanwhile, researchers in fields as varied as stylistics, neuropathology, language acquisition, and even forensics continue to use flawed LD indices — often ignorant that their results are questionable and in some cases potentially dangerous. Recently, an LD measurement instrument known as vocd has become the virtual tool of the LD trade. In this paper, we report both theoretical and empirical evidence that calls into question the rationale for vocd and also indicates that its reliability is not optimal. Although our evidence shows that vocd's output (D) is a relatively robust indicator of the aggregate probabilities of word occurrences in a text, we show that these probabilities — and thus also D — are affected by text length. Malvern, Richards, Chipere and Durán (2004) acknowledge that D (as calculated by vocd's default method) can be affected by text length, but claim that the effects are not significant for the ranges of text lengths with which they are concerned. In this paper, we explain why D is affected by text length, and demonstrate with an extensive empirical analysis that the effects of text length are significant over certain ranges, which we identify.

The Length and Semantic Structure of Article Titles—Evolving Disciplinary Practices and Correlations with Impact

“…It is defined as the number of unique phrases appearing in titles of a statistically large unit quota of literature (thousands of articles). It is similar to the lexical diversity used in computational linguistics to study richness of verbal expression (McKee et al, 2000;McCarthy and Jarvis, 2007;Koizumi, 2012), but uses phrases (concepts) instead of individual words. Bodies of literature that have more diverse concepts in titles will have a higher fraction of unique phrases and could be considered to cover larger cognitive extents.…”

MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment

“…But McCarthy and Jarvis (2007) demonstrated that the vocd-D value is actually based on probabilities of word occurrence and that D serves no purposes except to convert the LD value into a new scale. More specifically, McCarthy and Jarvis demonstrated that vocd-D is merely a complex way of approximating the hypergeometric distribution, and to demonstrate this, they described an index that we refer to here as HD-D.…”

“…For this reason, it is not surprising that McCarthy and Jarvis (2007) found correlations of r .971 between vocd-D and HD-D (i.e., sums of probabilities) for sample sizes from 35 to 50 (i.e., the sizes of samples that vocd-D uses in its random-sampling procedures). The correlation would have been perfect had it not been for the slight imprecision in vocd-D's output brought about by its reliance strung together.…”

“…Because vocd-D has been extensively described elsewhere, we offer here only a brief review of the approach and direct the interested reader to Malvern et al (2004) and McCarthy and Jarvis (2007). Furthermore, we discuss vocd-D using its system's default options-that is, those options selected by vocd-D's creators as being the best working parameters and the parameters that are typically reported in vocd-D studies.…”

Analysing lexical richness in French learner language: what frequency lists and teacher judgements can tell us about basic and advanced words

“…We compare different operationalisations of the AG with a well-known measure of lexical diversity, D (Malvern et al, 2004), which represents the single parameter of a mathematical function that models the falling TTR curve (see also Jarvis, 2002 andMcCarthy andJarvis, 2007 for an appraisal of this measure). The different measures can give us an indication to what extent the students from the three groups differ from each other in the quantity and/or in the quality of the vocabulary they use.…”