Identifying the Authors’ National Variety of English in Social Media Texts

Simaki, Vasiliki; Simakis, Panagiotis; Paradis, Carita; Kerren, Andreas

doi:10.26615/978-954-452-049-6_086

Cited by 14 publications

(19 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Elfardy and Diab (2013) used emoticons (or emojis) in Arabic dialect identification with Naive Bayes ("NB"; see Section 6.5). Non-alphabetic characters have also been used by Basile et al (2017), Bestgen (2017), Samih (2017), and Simaki et al (2017). Henrich (1989) used knowledge of alphabets to exclude languages where a language-unique character in a test document did not appear.…”

Section: Charactersmentioning

confidence: 99%

“…Capitalization Capitalization is mostly preserved when calculating character n-gram frequencies, but in contexts where it is possible to identify the orthography of a given document and where capitalization exists in the orthography, lowercasing can be used to reduce sparseness. In recent LI work, capitalization was used as a special feature by Basile et al (2017), Bestgen (2017), and Simaki et al (2017).…”

Section: Alphabetsmentioning

confidence: 99%

“…Nobesawa and Tahara (2005) used the length of full person names comprising several words. Lately, the number of characters in words has been used for LI by Dongen (2017), van der Lee and Bosch (2017), Samih (2017), and Simaki et al (2017). Dongen (2017) also used the length of the two preceding words.…”

Section: Alphabetsmentioning

confidence: 99%

“…Recently, the relative frequency of words has also been used for LI by Jauhiainen et al (2017b,a) and Jourlin (2017) Short words Grefenstette (1995) used the relative frequency of words with less than six characters. Recently, Panich (2015) also used short words, as did Simaki et al (2017). Alex (2005) used the relative frequency calculated from Google searches.…”

Section: Most Common Wordsmentioning

confidence: 99%

“…Sentence length used the number of words in a sentence with NB. van der Lee and Bosch (2017) and Simaki et al (2017) used the sentence length calculated in both words and characters with several machine learning algorithms. (2017) used the ratio to the total number of words of: once-occurring words, twice-occurring words, short words, long words, function words, adjectives and adverbs, personal pronouns, and question words.…”

Section: Word Combinationsmentioning

confidence: 99%

See 4 more Smart Citations

Automatic Language Identification in Texts: A Survey

Jauhiainen

Lui²,

Zampieri³

et al. 2019

jair

102

View full text Add to dashboard Cite

Language identification ("LI") is the problem of determining the natural language that a document or part thereof is written in. Automatic LI has been extensively researched for over fifty years. Today, LI is a key part of many text processing pipelines, as text processing techniques generally assume that the language of the input text is known. Research in this area has recently been especially active. This article provides a brief history of LI research, and an extensive survey of the features and methods used in the LI literature. We describe the features and methods using a unified notation, to make the relationships between methods clearer. We discuss evaluation methods, applications of LI, as well as off-the-shelf LI systems that do not require training by the end user. Finally, we identify open issues, survey the work to date on each issue, and propose future directions for research in LI.LI as a task predates computational methods -the earliest interest in the area was motivated by the needs of translators, and simple manual methods were developed to quickly identify documents in specific languages. The earliest known work to describe a functional LI program for text is by Mustonen (1965), a statistician, who used multiple discriminant analysis to teach a computer how to distinguish, at the word level, between English, Swedish and Finnish. Mustonen compiled a list of linguistically-motivated character-based features, and trained his language identifier on 300 words for each of the three target languages. The training procedure created two discriminant functions, which were tested with 100 words for each language. The experiment resulted in 76% of the words being correctly classified; even by current standards this percentage would be seen as acceptable given the small amount of training material, although the composition of training and test data is not clear, making the experiment unreproducible.In the early 1970s, Nakamura (1971) considered the problem of automatic LI. According to Rau (1974) and the available abstract of Nakamura's article, 1 his language identifier was able to distinguish between 25 languages written with the Latin alphabet. As features, the method used the occurrence rates of characters and words in each language. From the abstract it seems that, in addition to the frequencies, he used some binary presence/absence features of particular characters or words, based on manual LI. Rau (1974) wrote his master's thesis "Language Identification by Statistical Analysis" for the Naval Postgraduate School at Monterey, California. The continued interest and the need to use LI of text in military intelligence settings is evidenced by the recent articles of, for example, Rafidha Rehiman et al. (2013), Rowe et al. (2013), and Voss et al. (2014. As features for LI, Rau (1974) used, e.g., the relative frequencies of characters and character bigrams. With a majority vote classifier ensemble of seven classifiers using Kolmogor-Smirnov's Test of Goodness of Fit and Yule's characteristic (K), he managed...

show abstract

Section: Charactersmentioning

confidence: 99%

Section: Alphabetsmentioning

confidence: 99%

Section: Alphabetsmentioning

confidence: 99%

Section: Most Common Wordsmentioning

confidence: 99%

Section: Word Combinationsmentioning

confidence: 99%

See 3 more Smart Citations

Automatic Language Identification in Texts: A Survey

Jauhiainen

Lui²,

Zampieri³

et al. 2019

jair

102

View full text Add to dashboard Cite

show abstract

Specific Challenges of Variation and Text Types

Jauhiainen,

Zampieri,

Baldwin

et al. 2024

Synthesis Lectures on Human Language Technologies

View full text Add to dashboard Cite

Natural language processing for similar languages, varieties, and dialects: A survey

Zampieri

Nakov²,

Scherrer

2020

Nat. Lang. Eng.

View full text Add to dashboard Cite

There has been a lot of recent interest in the natural language processing (NLP) community in the computational processing of language varieties and dialects, with the aim to improve the performance of applications such as machine translation, speech recognition, and dialogue systems. Here, we attempt to survey this growing field of research, with focus on computational methods for processing similar languages, varieties, and dialects. In particular, we discuss the most important challenges when dealing with diatopic language variation, and we present some of the available datasets, the process of data collection, and the most common data collection strategies used to compile datasets for similar languages, varieties, and dialects. We further present a number of studies on computational methods developed and/or adapted for preprocessing, normalization, part-of-speech tagging, and parsing similar languages, language varieties, and dialects. Finally, we discuss relevant applications such as language and dialect identification and machine translation for closely related languages, language varieties, and dialects.

show abstract

Identifying the Authors’ National Variety of English in Social Media Texts

Cited by 14 publications

References 26 publications

Automatic Language Identification in Texts: A Survey

Automatic Language Identification in Texts: A Survey

Specific Challenges of Variation and Text Types

Natural language processing for similar languages, varieties, and dialects: A survey

Contact Info

Product

Resources

About