Comparative Evaluation of String Similarity Measures for Automatic Language Classification

Abstract

Historical linguistics, the oldest branch of modern linguistics, deals with language-relatedness and language change across space and time. Historical linguists apply the widely-tested comparative method [Durie and Ross, 1996] to establish relationships between languages to posit a language family and to reconstruct the proto-language for a language family. 1Although historical linguistics has parallel origins with biology [Atkinson and Gray, 2005], unlike the biologists, mainstream historical linguists have seldom been enthusiastic about using quantitative methods for the discovery of language relationships or investigating the structure of a language family, except for Kroeber and Chrétien [1937] and Ellegård [1959].A short period of enthusiastic application of quantitative methods initiated by Swadesh [1950] ended with the heavy criticism levelled against it by Bergsland and Vogt [1962]. The field of computational historical linguistics did not receive much attention again until the beginning of the 1990s, with the exception of two noteworthy doctoral dissertations, by Sankoff [1969] and Embleton [1986].In traditional lexicostatistics, as introduced by Swadesh [1952], distances between languages are based on human expert cognacy judgments of items in standardized word lists, e.g., the Swadesh lists [Swadesh, 1955]. In the terminology of historical linguistics, cognates are related words across languages that can be traced directly back to the proto-language.Cognates are identified through regular sound correspondences. Sometimes cognates have similar surface form and related meanings. Examples of such revealing kind of cognates are: English German∼ night ∼ Nacht 'night' and hound ∼ Hund 'dog'. If a word has undergone many changes then the relatedness is not obvious from visual inspection and one needs to look into the history of the word to exactly understand the sound changes which resulted in the synchronic form. For instance, the English Hindi ∼ wheel ∼ chakra 'wheel' are cognates and can be traced back to the proto-Indo-European root k w ek w lo-.

Mechanising Historical Phonology

“…The comparison of possible cognates initiated by Gleason and Kay has continued in various ways to the present, and the comparison of languages by automated 'cognate' recognition remains a growing area of research, with multiple methods in use, including the comparison of entire dictionaries (Rama & Borin 2015;List et al 2017;Rama et al 2017). Many of these studies, driven by the needs of machine translation, count as 'cognates' pairs like English nature : French nature as well as Indo-European cognates such as English brother : French fr ere, which have proved methodologically challenging (Inkpen et al 2005).…”

Revisiting the classification of Gallo-Italic: a dialectometric approach

“…Nevertheless, the current study also includes the Swadesh wordlists since these have been widely used in the literature on quantitative comparison of the major Romance and other Indo-European languages (e.g. Forster, Toth, & Bandelt, 1998;McMahon & McMahon, 2003;Rama et al, 2015).…”

Advances in Dialectometry

“…Although the use of an edit-distance measure in historical linguistics seems to have developed independently (Wichmann et al 2010), there is now an active exchange of ideas on the best refinements supporting historical inference (List 2012, Jäger 2014, Rama & Borin 2015. All of these papers explore enhancements of edit distance in which the cost of replacing a sound with another is systematically lower when the two sounds are phonetically Multidimensional scaling (MDS): a dimension-reduction technique for distance matrices similar.…”