Determining language variant in microblog messages

Laboreiro, Gustavo; Bošnjak, Matko; Sarmento, Luís; Rodrigues, Eduarda Mendes; Oliveira, Eugénio

doi:10.1145/2480362.2480535

Cited by 11 publications

(7 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The POS-tag patterns were then used by a neural network to indicate whether the sentences were written in Malay or not. Laboreiro et al (2013) used Jspell to detect differences in the grammar of Portuguese variants. Bekavac et al (2014) used a syntactic grammar to recognize verb-da-verb constructions, which are characteristic of the Serbian language.…”

Section: Statistics Of Words Van Der Lee and Boschmentioning

confidence: 99%

Automatic Language Identification in Texts: A Survey

Jauhiainen

Lui²,

Zampieri³

et al. 2019

jair

102

View full text Add to dashboard Cite

Language identification ("LI") is the problem of determining the natural language that a document or part thereof is written in. Automatic LI has been extensively researched for over fifty years. Today, LI is a key part of many text processing pipelines, as text processing techniques generally assume that the language of the input text is known. Research in this area has recently been especially active. This article provides a brief history of LI research, and an extensive survey of the features and methods used in the LI literature. We describe the features and methods using a unified notation, to make the relationships between methods clearer. We discuss evaluation methods, applications of LI, as well as off-the-shelf LI systems that do not require training by the end user. Finally, we identify open issues, survey the work to date on each issue, and propose future directions for research in LI.LI as a task predates computational methods -the earliest interest in the area was motivated by the needs of translators, and simple manual methods were developed to quickly identify documents in specific languages. The earliest known work to describe a functional LI program for text is by Mustonen (1965), a statistician, who used multiple discriminant analysis to teach a computer how to distinguish, at the word level, between English, Swedish and Finnish. Mustonen compiled a list of linguistically-motivated character-based features, and trained his language identifier on 300 words for each of the three target languages. The training procedure created two discriminant functions, which were tested with 100 words for each language. The experiment resulted in 76% of the words being correctly classified; even by current standards this percentage would be seen as acceptable given the small amount of training material, although the composition of training and test data is not clear, making the experiment unreproducible.In the early 1970s, Nakamura (1971) considered the problem of automatic LI. According to Rau (1974) and the available abstract of Nakamura's article, 1 his language identifier was able to distinguish between 25 languages written with the Latin alphabet. As features, the method used the occurrence rates of characters and words in each language. From the abstract it seems that, in addition to the frequencies, he used some binary presence/absence features of particular characters or words, based on manual LI. Rau (1974) wrote his master's thesis "Language Identification by Statistical Analysis" for the Naval Postgraduate School at Monterey, California. The continued interest and the need to use LI of text in military intelligence settings is evidenced by the recent articles of, for example, Rafidha Rehiman et al. (2013), Rowe et al. (2013), and Voss et al. (2014. As features for LI, Rau (1974) used, e.g., the relative frequencies of characters and character bigrams. With a majority vote classifier ensemble of seven classifiers using Kolmogor-Smirnov's Test of Goodness of Fit and Yule's characteristic (K), he managed...

show abstract

Section: Statistics Of Words Van Der Lee and Boschmentioning

confidence: 99%

Automatic Language Identification in Texts: A Survey

Jauhiainen

Lui²,

Zampieri³

et al. 2019

jair

102

View full text Add to dashboard Cite

show abstract

“…However, Brazil has ∼20 times the population of Portugal, which may explain Brazil having a greater number of publications in the research area. Therefore, if we analyse data from Twitter, one of the most widely used micro‐blogs for text mining, choosing a random tweet in Portuguese, there is a 95% chance of it originating in Brazil [29]. Facebook and Twitter are important sources of UGC; however, the former is less used in text mining as it often contains non‐text data, e.g.…”

Section: Discussionmentioning

confidence: 99%

“…Therefore, if we analyse data Others: Co-training algorithm, behaviour knowledge space, blocking doubtful predictions, conditional random fields, Dijkstra's shortest-path algorithm, entropy maximisation, expectation-maximisation, filtered space saving algorithm, Hill climbing algorithm, Hoeffding adaptive trees, active classifier, incremental lazy associative classifier, incremental model maintenance, JRip, Kaldor-Hicks-efficient selective sampling, linear regression, LocalMaxs algorithm, logic programming, logistic regression, map-reduce paradigm, noise-contrastive estimation, OneR classification algorithm, Online rule extraction, Pareto-efficient selective sampling, structural risk minimisation. from Twitter, one of the most widely used micro-blogs for text mining, choosing a random tweet in Portuguese, there is a 95% chance of it originating in Brazil [29]. Facebook and Twitter are important sources of UGC; however, the former is less used in text mining as it often contains non-text data, e.g.…”

Section: Main Findingsmentioning

confidence: 99%

Characterising text mining: a systematic mapping review of the Portuguese language

et al. 2018

View full text Add to dashboard Cite

Documents written in natural language constitute a major part of the artefacts produced during the software engineering life cycle. Studies indicate that more than 80% of enterprise data is stored in some sort of unstructured form, mainly as text. Therefore, the growth of user-generated content, especially from social media, provides a huge amount of data which allows discovering the experiences, opinions, and feelings of users. Text mining refers to the set of tools, techniques, and algorithms adopted to extract useful information from unstructured data. Considering that Portuguese ranks among the ten most spoken languages, and it is the second most common in Twitter, this study aims to map current primary studies that relate to the application of text mining for Portuguese. A systematic mapping method was applied and 6075 primary studies were retrieved up to the year 2014. A total of 203 studies were included, from which more than 60% analyse texts written in Brazilian variant. The majority of studies focus on the text classification task. Support vector machine and Naïve Bayes appear as main the algorithms. Folha de São Paulo and Público newspapers appear as main corpora, followed by the Portuguese Attorney General's Office corpus and Twitter.

show abstract

“…Laboreiro et al [35] used a Bayesian classifier to distinguish between European and Brazilian variants of tweets written in Portuguese language, achieving 95% accuracy. Winkelmolen and Mascardi [72] also describe a Bayesian classifier that performs well on very short texts and made experiments on film subtitles in 22 languages.…”

Section: Tweets/short Messagesmentioning

confidence: 99%

“…However, the emergence of social media and the chatspeak employed by its users has brought about new previously unseen issues that need to be studied in order to deal with these kinds of texts. Three key issues posited in the literature [63,24,69] and that, as of today, cannot be considered solved include: (i) distinguishing similar languages [76], (ii) dealing with multilingual documents [43], and (iii) language identification for short texts [6,10,35,20,70,52]. The shared task organized at TweetLID has considered these three unresolved issues, and has enabled participants to compare the performance of their systems in these situations.…”

Section: Challengesmentioning

confidence: 99%

TweetLID: a benchmark for tweet language identification

Zubiaga

Vicente²,

Gamallo

et al. 2015

Lang Resources & Evaluation

View full text Add to dashboard Cite

Language identification, as the task of determining the language a given text is written in, has progressed substantially in recent decades. However, three main issues remain still unresolved: (i) distinction of similar languages, (ii) detection of multilingualism in a single document, and (iii) identifying the language of short texts. In this paper, we describe our work on the development of a benchmark to encourage further research in these three directions, set forth an evaluation framework suitable for the task, and make a dataset of annotated tweets publicly available for research purposes. We also describe the shared task we organized to validate and assess the evaluation framework and dataset with systems submitted by seven different participants, and analyze the performance of these systems. The evaluation of the results submitted by the participants of the shared task helped us shed some light on the shortcomings of state-of-the-art language identification systems, and gives insight into the extent to which the brevity, multilingualism, and language similarity found in texts exacerbate the performance of language identifiers. Our dataset with nearly 35,000 tweets and the evaluation framework provide researchers and practitioners with suitable resources to further study the aforementioned issues on language identification within a common setting that enables to compare results with one another.

show abstract

Determining language variant in microblog messages

Cited by 11 publications

References 11 publications

Automatic Language Identification in Texts: A Survey

Automatic Language Identification in Texts: A Survey

Characterising text mining: a systematic mapping review of the Portuguese language

TweetLID: a benchmark for tweet language identification

Contact Info

Product

Resources

About