Dialectometric analysis of language variation in Twitter

Abstract

In the last few years, microblogging platforms such as Twitter have given rise to a deluge of textual data that can be used for the analysis of informal communication between millions of individuals. In this work, we propose an informationtheoretic approach to geographic language variation using a corpus based on Twitter. We test our models with tens of concepts and their associated keywords detected in Spanish tweets geolocated in Spain. We employ dialectometric measures (cosine similarity and Jensen-Shannon divergence) to quantify the linguistic distance on the lexical level between cells created in a uniform grid over the map. This can be done for a single concept or in the general case taking into account an average of the considered variants. The latter permits an analysis of the dialects that naturally emerge from the data. Interestingly, our results reveal the existence of two dialect macrovarieties. The first group includes a region-specific speech spoken in small towns and rural areas whereas the second cluster encompasses cities that tend to use a more uniform variety. Since the results obtained with the two different metrics qualitatively agree, our work suggests that social media corpora can be efficiently used for dialectometric analyses.

Global Syntactic Variation in Seven Languages: Toward a Computational Dialectology

“…Even though these international languages have global speech communities, dialectology and sociolinguistics continue to focus largely on sub-national dialects, often within so-called inner-circle varieties (Kachru, 1982). This paper joins recent work in taking a global approach by using geo-referenced texts (Goldhahn et al, 2012;Davies and Fuchs, 2015;Donoso and Sanchez, 2017) to represent national varieties (Szmrecsanyi et al, 2016;Calle-Martin and Romero-Barranco, 2017;Cook and Brinton, 2017;Rangel et al, 2017;Dunn, 2018aDunn, , 2019bTamaredo, 2018). The basic point is that in order to represent regional variation as a complete system, dialectometry must take a global perspective.…”

Time of Your Hate: The Challenge of Time in Hate Speech Detection on Social Media

“…The recent availability of long-term and large-scale digital corpora and the effectiveness of methods for representing words over time played a crucial role in the recent advances in this field. However, only a few attempts focused on social media [28,29], and their goal is to analyze linguistic aspects rather than understanding how lexical semantic change can affect performance in sentiment analysis or hate speech detection. From this perspective, our work represents a novelty: for the first time, we propose to tackle the issue of diachronic degradation of hate speech detection by exploring the temporal robustness of prediction models.…”

Mapping languages: the Corpus of Global Language Use

“…In isolation, web-crawled data provides a single observation of digital language use. Another common source of data is from Twitter (e.g., Eisenstein, et al, 2010;Roller, et al, 2012;Kondor, et al, 2013;Mocanu, et al, 2013;Eisenstein, et al, 2014;Graham, et al, 2014;Donoso & Sanchez, 2017). This paper uses a baseline Twitter corpus as a point of comparison: does the Common Crawl agree with Twitter data?…”