Lexical Normalization for Code-switched Data and its Effect on POS Tagging

Goot, Rob van der; Çetinoğlu, Özlem

doi:10.18653/v1/2021.eacl-main.200

Cited by 11 publications

(3 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Indeed, in our experience speakers write ''the way words sound'' in their local variants, using just the available characters in their keyboards. Normalizing user-generated texts to a ''standard'' form (e.g., Baldwin et al, 2015;van der Goot et al, 2020van der Goot et al, , 2021a has proven useful for NLP purposes, but it inevitably erases the naturally occurring sociolinguistic variation (Nguyen et al, 2021), homogenizing all variants of a language variety and imposing a ''correct'' form of writing.…”

Section: Uniform Functions Contexts and Needsmentioning

confidence: 99%

Language Varieties of Italy: Technology Challenges and Opportunities

Ramponi

2024

Transactions of the Association for Computational Linguistics

View full text Add to dashboard Cite

Italy is characterized by a one-of-a-kind linguistic diversity landscape in Europe, which implicitly encodes local knowledge, cultural traditions, artistic expressions, and history of its speakers. However, most local languages and dialects in Italy are at risk of disappearing within a few generations. The NLP community has recently begun to engage with endangered languages, including those of Italy. Yet, most efforts assume that these varieties are under-resourced language monoliths with an established written form and homogeneous functions and needs, and thus highly interchangeable with each other and with high-resource, standardized languages. In this paper, we introduce the linguistic context of Italy and challenge the default machine-centric assumptions of NLP for Italy’s language varieties. We advocate for a shift in the paradigm from machine-centric to speaker-centric NLP, and provide recommendations and opportunities for work that prioritizes languages and their speakers over technological advances. To facilitate the process, we finally propose building a local community towards responsible, participatory efforts aimed at supporting vitality of languages and dialects of Italy.

show abstract

Section: Uniform Functions Contexts and Needsmentioning

confidence: 99%

Language Varieties of Italy: Technology Challenges and Opportunities

Ramponi

2024

Transactions of the Association for Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…The dataset is available through a GitHub repository. Besides these monolingual resources, a normalization dataset for Turkish-German is also available (van der Goot and Çetinoğlu, 2021). This dataset is a revised version of the data from Çetinoğlu and Çöltekin (2016) for normalization by employing token-level alignment layers and adapting existing language ID and POS tags for these new layers.…”

Section: Social Media Text Normalization Corporamentioning

confidence: 99%

Resources for Turkish Natural Language Processing: A critical survey

Çöltekin¹,

Doğruöz²,

Çetinoğlu³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

This paper presents a comprehensive survey of corpora and lexical resources available for Turkish. We review a broad range of resources, focusing on the ones that are publicly available. In addition to providing information about the available linguistic resources, we present a set of recommendations, and identify gaps in the data available for conducting research and building applications in Turkish Linguistics and Natural Language Processing.

show abstract

“…Avrupa ve Amerika'da Valentina Day diye geçer. Turkish-German (van der Goot and Çetinoglu, 2021) artik ablamdan bise yuruturum napim :D Artık ablamdan bir şey yürütürüm ne yapayım :D Table 2: Noisy examples from each language and the corresponding canonical forms.…”

Section: Language Example Raw Example Goldunclassified

Sequence-to-Sequence Lexical Normalization with Multilingual Transformers

Bucur¹,

Cosma²,

Dinu³

2021

Preprint

View full text Add to dashboard Cite

Current benchmark tasks for natural language processing contain text that is qualitatively different from the text used in informal day to day digital communication. This discrepancy has led to severe performance degradation of state-of-the-art NLP models when fine-tuned on real-world data. One way to resolve this issue is through lexical normalization, which is the process of transforming non-standard text, usually from social media, into a more standardized form. In this work, we propose a sentence-level sequence-to-sequence model based on mBART, which frames the problem as a machine translation problem. As the noisy text is a pervasive problem across languages, not just English, we leverage the multilingual pre-training of mBART to fine-tune it to our data. While current approaches mainly operate at the word or subword level, we argue that this approach is straightforward from a technical standpoint and builds upon existing pre-trained transformer networks. Our results show that while word-level, intrinsic, performance evaluation is behind other methods, our model improves performance on extrinsic, downstream tasks through normalization compared to models operating on raw, unprocessed, social media text.

show abstract

Lexical Normalization for Code-switched Data and its Effect on POS Tagging

Cited by 11 publications

References 23 publications

Language Varieties of Italy: Technology Challenges and Opportunities

Language Varieties of Italy: Technology Challenges and Opportunities

Resources for Turkish Natural Language Processing: A critical survey

Sequence-to-Sequence Lexical Normalization with Multilingual Transformers

Contact Info

Product

Resources

About