On the Accuracy of Language Trees

Abstract

Historical linguistics aims at inferring the most likely language phylogenetic tree starting from information concerning the evolutionary relatedness of languages. The available information are typically lists of homologous (lexical, phonological, syntactic) features or characters for many different languages: a set of parallel corpora whose compilation represents a paramount achievement in linguistics.From this perspective the reconstruction of language trees is an example of inverse problems: starting from present, incomplete and often noisy, information, one aims at inferring the most likely past evolutionary history. A fundamental issue in inverse problems is the evaluation of the inference made. A standard way of dealing with this question is to generate data with artificial models in order to have full access to the evolutionary process one is going to infer. This procedure presents an intrinsic limitation: when dealing with real data sets, one typically does not know which model of evolution is the most suitable for them. A possible way out is to compare algorithmic inference with expert classifications. This is the point of view we take here by conducting a thorough survey of the accuracy of reconstruction methods as compared with the Ethnologue expert classifications. We focus in particular on state-of-the-art distance-based methods for phylogeny reconstruction using worldwide linguistic databases.In order to assess the accuracy of the inferred trees we introduce and characterize two generalizations of standard definitions of distances between trees. Based on these scores we quantify the relative performances of the distance-based algorithms considered. Further we quantify how the completeness and the coverage of the available databases affect the accuracy of the reconstruction. Finally we draw some conclusions about where the accuracy of the reconstructions in historical linguistics stands and about the leading directions to improve it.

Are Automatic Methods for Cognate Detection Good Enough for Phylogenetic Reconstruction in Historical Linguistics?

“…All our experiments are performed using MrBayes 3.2.6 (Zhang et al, 2015). Pompei et al (2011) introduced Generalized Quartet Distance (GQD) as an extension to Quartet Distance (QD) in order to compare binary trees with a polytomous tree, since gold standard trees can have non-binary internal nodes. It was widely used for comparing inferred language phylogenies with gold standard phylogenies (Greenhill et al, 2010;Wichmann et al, 2011;Jäger, 2013).…”

Cross-Language Distributions of High Frequency and Phonetically Similar Cognates

“…This normalized Levenshtein Distance (NLD) makes sure that identical form-overlap between translation equivalents results in similarity scores of one, and no overlap results in a score of zero. Various slightly different ways to normalize the Levenshtein distance have been utilized, see Pompei et al [40] for a comparison. Mackay and Kondrak [17] have argued that the Levenshtein distance needs to be normalized in an exponential instead of in a linear way.…”

New methodologies for historical linguistics?

“…Elements of Gray et al 's study are representative of similar works (e.g., Dunn et al 2008), and our rubric of evaluation should serve as a test for them as well (e.g. Donohue 2012; see Pompei et al 2011 for alternative evaluation metrics). Gray et al (2009, cf.…”