On the origin of long-range correlations in texts

Abstract

The complexity of human interactions with social and natural phenomena is mirrored in the way we describe our experiences through natural language. In order to retain and convey such a high dimensional information, the statistical properties of our linguistic output has to be highly correlated in time. An example are the robust observations, still largely not understood, of correlations on arbitrary long scales in literary texts. In this paper we explain how long-range correlations flow from highly structured linguistic levels down to the building blocks of a text (words, letters, etc..). By combining calculations and data analysis we show that correlations take form of a bursty sequence of events once we approach the semantically relevant topics of the text. The mechanisms we identify are fairly general and can be equally applied to other hierarchical settings.complex systems | language dynamics | long correlations | statistical physics | burstiness L iterary texts are an expression of the natural language ability to project complex and high-dimensional phenomena into a one-dimensional, semantically meaningful sequence of symbols. For this projection to be successful, such sequences have to encode the information in form of structured patterns, such as correlations on arbitrarily long scales (1, 2). Understanding how language processes long-range correlations, an ubiquitous signature of complexity present in human activities (3-7) and in the natural world (8-11), is an important task towards comprehending how natural language works and evolves. This understanding is also crucial to improve the increasingly important applications of information theory and statistical natural language processing, which are mostly based on short-range-correlations methods (12-15).Take your favorite novel and consider the binary sequence obtained by mapping each vowel into a 1 and all other symbols into a 0. One can easily detect structures on neighboring bits, and we certainly expect some repetition patterns on the size of words. But one should certainly be surprised and intrigued when discovering that there are structures (or memory) after several pages or even on arbitrary large scales of this binary sequence. In the last twenty years, similar observations of long-range correlations in texts have been related to large scales characteristics of the novels such as the story being told, the style of the book, the author, and the language (1, 2, 16-21). However, the mechanisms explaining these connections are still missing (see ref. 2 for a recent proposal). Without such mechanisms, many fundamental questions cannot be answered. For instance, why all previous investigations observed long-range correlations despite their radically different approaches? How and which correlations can flow from the high-level semantic structures down to the crude symbolic sequence in the presence of so many arbitrary influences? What information is gained on the large structures by looking at smaller ones? Finally, what is the origin of the longrange c...

Evaluating the Irregularity of Natural Languages

“…When applied to natural language, our results show that the temporal organization of natural languages (with some differences between them) exhibits more complex structure than the sequences constructed by randomizations. These results are also concordant with previous studies, which report the presence of long-range correlations in written texts [33,34]. …”

Scarcity of crossing dependencies: A direct outcome of a specific constraint?

“…However, real sentences are not random sequences of words, as research on long correlations in physics has been showing for more than a decade, e.g. [61,62]. Second, although such null model predictor has been tested previously on uniformly random trees [28], one cannot assume the predictor will work on real sentences given the substantial statistical differences between uniformly random trees and real syntactic dependency trees [18].…”

Critical Behavior in Physics and Probabilistic Formal Languages

“…It has important applications beyond the traditional purview of physics, as well [1][2][3][4][5], including applications to music [4,6], genomics [7,8] and human languages [9][10][11][12].…”