Context tree selection and linguistic rhythm retrieval from written texts

Abstract

The starting point of this article is the question "How to retrieve fingerprints of rhythm in written texts?" We address this problem in the case of Brazilian and European Portuguese. These two dialects of Modern Portuguese share the same lexicon and most of the sentences they produce are superficially identical. Yet they are conjectured, on linguistic grounds, to implement different rhythms. We show that this linguistic question can be formulated as a problem of model selection in the class of variable length Markov chains. To carry on this approach, we compare texts from European and Brazilian Portuguese. These texts are previously encoded according to some basic rhythmic features of the sentences which can be automatically retrieved. This is an entirely new approach from the linguistic point of view. Our statistical contribution is the introduction of the smallest maximizer criterion which is a constant free procedure for model selection. As a by-product, this provides a solution for the problem of optimal choice of the penalty constant when using the BIC to select a variable length Markov chain. Besides proving the consistency of the smallest maximizer criterion when the sample size diverges, we also make a simulation study comparing our approach with both the standard BIC selection and the Peres-Shields order estimation. Applied to the linguistic sample constituted for our case study, the smallest maximizer criterion assigns different context-tree models to the two dialects of Portuguese. The features of the selected models are compatible with current conjectures discussed in the linguistic literature.

Consistent Estimation of Partition Markov Models

“…The distance allows the construction of consistent estimation algorithms to identify the partition. Research in progress suggests that this measure can be harnessed for the development and implementation of robust estimation techniques, given that there are records (see [19]) of the need of these techniques for Markov processes. In summary, in this paper, in addition to responding positively to the question of whether the Bayesian Information Criterion is capable of allowing a consistent estimation of the partition of the Markov process, we also obtain that, in terms of the model selection procedure, the Bayesian Information Criterion corresponds to a distance in the state space of the Markov process.…”

Classifying Written Texts Through Rhythmic Features

“…The rhythm of language appears to be culturally regulated. Galves et al [10] extract streams of stresses from corpora of newspaper articles written in both European and Brazilian Portuguese, and use Variable Length Markov Chains [11] to model rhythmic realization in the two corpora, arriving at different final models. Where cultural background influences linguistic rhythm, it similarly influences musical rhythm, as shown by Patel and Daniele [12].…”

Robust Model Selection for Stochastic Processes

“…There are diverse methodologies for VLMC model selection (see Rissanen, 1983;Buhlmann et al, 1999;Csiszár et al, 2006;Galves et al, 2012). This article uses version of the Context Tree Maximization (from now on CTM) algorithm introduced by Csiszár et al (2006), which is based on the Bayesian Information Criterion (BIC).…”

“…The strategy introduced in this article was applied in Garcia et al (2012) to linguistic data from eight languages. In Garcia et al (2012), the objective was to check a linguistic conjecture that classifies languages into three rhythmic classes: "syllable-timed," "stresstimed" and "mora-timed."…”