Hierarchical structures induce long-range dynamical correlations in written texts

Abstract

Thoughts and ideas are multidimensional and often concurrent, yet they can be expressed surprisingly well sequentially by the translation into language. This reduction of dimensions occurs naturally but requires memory and necessitates the existence of correlations, e.g., in written text. However, correlations in word appearance decay quickly, while previous observations of long-range correlations using random walk approaches yield little insight on memory or on semantic context. Instead, we study combinations of words that a reader is exposed to within a ''window of attention,'' spanning about 100 words. We define a vector space of such word combinations by looking at words that co-occur within the window of attention, and analyze its structure. Singular value decomposition of the co-occurrence matrix identifies a basis whose vectors correspond to specific topics, or ''concepts'' that are relevant to the text. As the reader follows a text, the ''vector of attention'' traces out a trajectory of directions in this ''concept space.'' We find that memory of the direction is retained over long times, forming power-law correlations. The appearance of power laws hints at the existence of an underlying hierarchical network. Indeed, imposing a hierarchy similar to that defined by volumes, chapters, paragraphs, etc. succeeds in creating correlations in a surrogate random text that are identical to those of the original text. We conclude that hierarchical structures in text serve to create long-range correlations, and use the reader's memory in reenacting some of the multidimensionality of the thoughts being expressed.hierarchy ͉ language ͉ power laws ͉ singular value decomposition L anguage is a central link through which we interact with other people. As a channel of communication it is limited by our physical ability to speak only one word at a time. The question arises therefore how the complex products of our brain are transformed into the linear string of words that comprise speech or text. Since our mental processes are far from being onedimensional, the use of memory is essential, as is the existence of some type of correlations in time.Such questions have a long and intense history. Bolzano (1) already noted the need for specific organization in scientific texts, while Ingarden devotes his book (2) to understanding the process by which a text is understood and assimilated. Modern methods (3, 4) combine the work of linguists with those of computer scientists, physicists, physiologists, and researchers from many other fields to cover a wide range of texts, from the phoneme (5), going on to words (6-9, ʈ) and grammar (10, 11), and all of the way to global text analysis (12) and the evolution of language (13,14).Recent interest has focused on applying methods of statistical physics to identify possible trends and correlations in text (15-18). In ref. 18, for example, the authors study the distribution of words across different works by the same authors, combining notions of information, entropy, and statistics to def...

Complexity and Universality in the Long-Range Order of Words

“…Although language sequences are not produced by a stochastic source, it is generally assumed that a large collection of language samples represent an ensemble with enough consistency in its statistical structure to allow the application of the standard formalism of information theory. However, one serious hurdle in computing the entropy of language based on the estimation of block probabilities is the presence of long-range correlations that span from hundreds to thousands of words [20][21][22][23][24]. The sample size that would be needed to estimate the required probabilities grows exponentially with block length, thus quickly rendering insufficient any any available linguistic source.…”

Long-range dependencies make the difference—Comment on “A stochastic model for EEG microstate sequence analysis”

“…Numerous studies demonstrate LRD for example in geological and climate research (Scheffer et al, 2009;Varotsos and Kirk-Davidoff, 2006), finance market fluctuations (Matteo et al, 2005;Robinson, 2003), in Internet modeling and network traffic analysis (Abry et al, 2002;Karagiannis et al, 2004;Riedi et al, 1999), or statistic analyses of human language (Alvarez-Lacalle et al, 2006;Petersen et al, 2012). Fig.…”

Records in fractal stochastic processes

“…In order to model turbulent flows, fractional Brownian motion (fBm), a generalization of the more well-known Brownian motion, was introduced many decades ago [41,42], and has become one of the most studied stochastic processes widely used in a variety of fields, including physics, probability, statistics, hydrology, economy, biology, and many others [43][44][45][46][47][48][49]. A fBm is a self-similar Gaussian process with stationary increments (called fractional Gaussian noise-fGn) and possesses long-range linear correlation which depends on a parameter, called the Hurst exponent, H [50], where 0 < H < 1.…”