Semi-automatic Discovery of Multilingual Elements in English Historical Corpora: Methods and Challenges

Abstract

One of the main obstacles standing in the way of large-scale studies of multilingual practices has been the difficulty of discovering secondary, i.e., foreign or minority language words by any other than manual means (see Pahta 2004;Nurmi and Pahta 2012). Although current computational methods for identifying the primary language of a monolingual text are robust (see, e.g., Alex, Dubey and Keller 2007), their accuracy diminishes significantly when the task is to identify short chunks of words and phrases in other languages within texts written predominantly in English (see King, Kübler and Hooper 2015). While established language-detection methods such as the use of foreign-language tags in POS-annotated corpora (see Grefenstette 1995), dictionarybased retrieval (see Alex 2005), and non-standard letter clusters (see Andersen 2012) can all be useful in word-level language identification, acceptable rates of both precision and recall can only be accomplished when the discovery logic is based on triangulation between multiple methods at the same time. Furthermore, when the objective is not only to identify passages that appear foreign based on form alone but also to distinguish between switches into another language and word sequences that have been assimilated into contemporary English usage in a particular register, the task becomes significantly more challenging.In this paper we discuss the semi-automatic method of discovering foreign-language passages developed by the Multilingual Practices in the History of Written English project 1 for the purpose of corpus-based analysis of multilingual practices in English historical texts (see Nurmi et al.forthcoming; Tyrkkö and Nurmi forthcoming). Using the Corpus of Late Modern English Texts 3.0