Code-Switching Ubique Est - Language Identification and Part-of-Speech Tagging for Historical Mixed Text

Abstract

In this paper, we describe the development of a language identification system and a part-of-speech tagger for Latin-Middle English mixed text. To this end, we annotate data with language IDs and Universal POS tags (Petrov et al., 2012). As a classifier, we train a conditional random field classifier for both sub-tasks, including features generated by the TreeTagger models of both languages. The focus lies on both a general and a task-specific evaluation. Moreover, we describe our effort concerning beyond proof-of-concept implementation of tools and towards a more task-oriented approach, showing how to apply our techniques in the context of Humanities research.

Detecting de minimis Code-Switching in Historical German Books

“…Besides this descriptive task, we are interested in practical tasks for predicting code-switching. There has been previous work formalizing code-switching detection in historical texts as a language ID task (Schulz and Keller, 2016;Sprugnoli et al, 2017), and models such as Conditional Random Fields (CRF) have been deployed to classify words as in one language or another. However, such approaches fail to work in the following scenario: when large collections of page images are transcribed with optical character recognition (OCR) or when large audio collections are transcribed by speech recognition, we do not always know a priori which languages will be included.…”

The limits of Spanglish?

“…Research into C-S in spontaneously-produced and elicited spoken speech has offered insights into the social, cognitive, and structural dimensions of this multilingual phenomenon (Bullock and Toribio, 2009). The analysis of C-S in written discourse has garnered substantially less attention and, with some exceptions reviewed below (Montes-Alcalá, 2001;Callahan, 2004Callahan, , 2002, it has centered largely on C-S in historical texts as a genre (Latin macaronic poetry, medieval Castilian Spanish-Hebrew taqqanots 'ordinances', personal letters) (Demo, 2018;Schulz and Keller, 2016;Miller, 2001;Gardner-Chloros and Weston, 2015;Swain et al, 2002;Nurmi and Pahta, 2004).…”

Proceedings of the 3rd Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature

“…Research into C-S in spontaneously-produced and elicited spoken speech has offered insights into the social, cognitive, and structural dimensions of this multilingual phenomenon (Bullock and Toribio, 2009). The analysis of C-S in written discourse has garnered substantially less attention and, with some exceptions reviewed below (Montes-Alcalá, 2001;Callahan, 2004Callahan, , 2002, it has centered largely on C-S in historical texts as a genre (Latin macaronic poetry, medieval Castilian Spanish-Hebrew taqqanots 'ordinances', personal letters) (Demo, 2018;Schulz and Keller, 2016;Miller, 2001;Gardner-Chloros and Weston, 2015;Swain et al, 2002;Nurmi and Pahta, 2004).…”