Complexity-entropy analysis at different levels of organisation in written language

Abstract

Written language is complex. A written text can be considered an attempt to convey a meaningful message which ends up being constrained by language rules, context dependence and highly redundant in its use of resources. Despite all these constraints, unpredictability is an essential element of natural language. Here we present the use of entropic measures to assert the balance between predictability and surprise in written text. In short, it is possible to measure innovation and context preservation in a document. It is shown that this can also be done at the different levels of organization of a text. The type of analysis presented is reasonably general, and can also be used to analyze the same balance in other complex messages such as DNA, where a hierarchy of organizational levels are known to exist.

Cancer Segmentation by Entropic Analysis of Ordered Gene Expression Profiles

“…We will be using plots of E vs. h , which is an example of what is called a complexity–entropy map. Complexity–entropy maps show the trade-off between randomness or pattern production as measured by h and pattern recurrence or memory as measured by E ; it has been used (under different names) in studies of an extensive set of systems [ 1 , 4 , 43 ].…”

“…Entropic magnitudes, as defined in information theory and related areas, have been used in a wide number of areas [ 1 ] beyond communication and physics, including literary analysis [ 2 , 3 , 4 ], painting [ 5 ], and music [ 6 ], among others. The advantage of entropic variables is the general framework on which they are founded, allowing them to be used in any system that conveys information in a broad sense.…”

Complexity and Entropy in Legal Language

“…[22]; additionally to a neural language model, utilized compression ratios to measure the complexity of the language used by the Supreme Courts of the U.S. (USSC) and Germany (BGH). [23]; using the Lempel-Ziv complexity measure C 76 , took into account not only the order inherent in a grammatically correct sentence, but also the larger organization of a text document, e.g. sections, by selectively shuffling the data belonging to each level of the hierarchy.…”

“…7, with a version of a statistical complexity measure, quantitatively study Shakespeare and other English Renaissance authors. [23]; used for the complexity-entropy plane the entropy rate and the entropy density and studied the organization of literary texts (Shakespeare, Abbott and Doyle) at different levels of the hierarchy. In order to calculate the entropy rate and density, which are asymptotic quantities, they used the Lempel-Ziv complexity C 76 .…”

Using word embeddings to improve the discriminability of co-occurrence text networks

“…Complex networks have been used in a wide range of fields, including in Social Sciences [11], Neuroscience [55], Biology [44], Scientometry [58] and Pattern Recognition [7,[12][13][14]. In text analysis, networks are used to uncover language patterns, including the origins of the ever present Zipf's Law [17] and the analysis of linguistic properties of natural and unknown texts [23,40]. Applications of network science in text mining and text classification encompasses applications in semantic analysis [21,28,52,53], authorship attribution [19,51] and stylometry [26,27,51].…”