Statistics and Machine Learning Experiments in English and Romanian Poetry

Abstract

This paper presents a quantitative approach to poetry, based on the use of several statistical measures (entropy, informational energy, N-gram, etc.) applied to a few characteristic English writings. We found that English language changes its entropy as time passes, and that entropy depends on the language used and on the author. In order to compare two similar texts, we were able to introduce a statistical method to asses the information entropy between two texts. We also introduced a method of computing the average information conveyed by a group of letters about the next letter in the text. We found a formula for computing the Shannon language entropy and we introduced the concept of N-gram informational energy of a poetry. We also constructed a neural network, which is able to generate Byron-type poetry and to analyze the information proximity to the genuine Byron poetry.

The combinatorial analysis of n-gram dictionaries, coverage and information entropy based on the web corpus of English

“…Despite the existence of other models, for example described by Chomsky (1956), many studies of natural languages, and in particular English, use the approximation of the text by the stationary Marcov process. For example, in the papers of Calin (2020), Hahn andSivley (2011), Yadav et al (2010), Guerrero (2009) the Marcov process is used to simulate a natural language text. Since the accuracy of approximating a natural language text using the Marcov model decreases significantly with an increase in the order of n-gram, related studies mainly investigate the entropy of short-length texts.…”