SeedLing: Building and Using a Seed corpus for the Human Language Project

Abstract

A broad-coverage corpus such as the Human Language Project envisioned by Abney and Bird (2010) would be a powerful resource for the study of endangered languages. Existing corpora are limited in the range of languages covered, in standardisation, or in machine-readability. In this paper we present SeedLing, a seed corpus for the Human Language Project. We first survey existing efforts to compile cross-linguistic resources, then describe our own approach. To build the foundation text for a Universal Corpus, we crawl and clean texts from several web sources that contain data from a large number of languages, and convert them into a standardised form consistent with the guidelines of Abney and Bird (2011). The resulting corpus is more easily-accessible and machine-readable than any of the underlying data sources, and, with data from 1451 languages covering 105 language families, represents a significant base corpus for researchers to draw on and add to in the future. To demonstrate the utility of SeedLing for cross-lingual computational research, we use our data in the test application of detecting similar languages.

Decoding Anagrammed Texts Written in an Unknown Language and Script

“…We now directly evaluate the three methods described above by applying them to a set of ciphertexts from different languages. We adapted the dataset created by Emerson et al (2014) from the text of the Universal Declaration of Human Rights (UDHR) in 380 languages. 2 The average length of the texts is 1710 words and 11073 characters.…”

USAAR-CHRONOS: Crawling the Web for Temporal Annotations

“…In this paper, we have described our submission to the Diachronic Text Evaluation for SemEval-2015. 5 The cleaning tool used is a compilation of web cleaning scripts (Emerson et al, 2014;Tan et al, 2014b;Tan and Bond, 2011) We have adapted a web crawler to search for the source of the text snippets used for the evaluation and achieved the highest precision score. Additionally, we have crawled and cleaned the source articles of the snippets and produced the Daikon corpus that can be used for future research in diachronic/temporal analysis and epoch identification.…”

Experimental Analysis of the Dorabella Cipher with Statistical Language Models

“…We chose Dangerous Connections, an English translation of an epistolary novel, for deriving character-level language models; and a much larger New York Times Corpus 5 for deriving word-level language models. For our language identification e xperiments, w e u se a d ataset constructed from 380 translations of the Universal Declaration of Human Rights (UDHR) (Emerson et al, 2014), and the multilingual OpenSubtitles corpus of movie subtitles (Lison and Tiedemann, 2016).…”