A Refutation of Finite-State Language Models through Zipf’s Law for Factual Knowledge

Abstract

We present a hypothetical argument against finite-state processes in statistical language modeling that is based on semantics rather than syntax. In this theoretical model, we suppose that the semantic properties of texts in a natural language could be approximately captured by a recently introduced concept of a perigraphic process. Perigraphic processes are a class of stochastic processes that satisfy a Zipf-law accumulation of a subset of factual knowledge, which is time-independent, compressed, and effectively inferrable from the process. We show that the classes of finite-state processes and of perigraphic processes are disjoint, and we present a new simple example of perigraphic processes over a finite alphabet called Oracle processes. The disjointness result makes use of the Hilberg condition, i.e., the almost sure power-law growth of algorithmic mutual information. Using a strongly consistent estimator of the number of hidden states, we show that finite-state processes do not satisfy the Hilberg condition whereas Oracle processes satisfy the Hilberg condition via the data-processing inequality. We discuss the relevance of these mathematical results for theoretical and computational linguistics.

Local Grammar-Based Coding Revisited

“…Seen in this light, Theorems 3 and 4 make a link between Hilberg's hypothesis about a power-law growth of mutual information for natural language [22,7] and Herdan-Heaps' law about the power-law growth of the number of distinct words in a text [19,17], which is a corollary of Zipf's law. For more details, see [10,11,12].…”

“…In particular, we may estimate the Markov order of empirical data using a consistent estimator, such as one introduced by Merhav, Gutman, and Ziv [25], and we may count the number of distinct substrings of the length equal to the Markov order estimate. In such case, we may observe that natural language contains many more such substrings than memoryless sources, see [11,12].…”

“…Consequently, we may naively suspect that the fully minimal code may be used for quantifying hierarchical structure in the stream of symbols generated by an information source [10], see also [8,27,28,29]. In particular, such a code might be used to explain power-law word frequency distributions from linguistics [32,19,17] in terms of a hypothetical power-law growth of mutual information [22,7], see [10,11,12] for more theory and [31,21,16,4,23,18,20] for more experimental data.…”

“…This phenomenon should be contrasted with a stark difference between natural language and memoryless sources for some other word-like segmentation procedures. Namely, if we estimate the Markov order of empirical data in a consistent way [25] and we count the number of distinct substrings of the length equal to the Markov order estimate then natural language exhibits many more such substrings than memoryless sources, see [11,12].…”