Part-of-speech tagging for English-Spanish code-switched text

Solorio, Thamar; Liu, Yang

doi:10.3115/1613715.1613852

Cited by 94 publications

(110 citation statements)

References 27 publications

Supporting

Mentioning

107

Contrasting

Unclassified

Order By: Relevance

“…This is also reflected in the abundant literature that uses Spanish HMMsbased taggers (e.g., [16][17][18][19], such as Schmid's (13) and Padró (20). In contrast, with a more recent history in language applications in general, MaxEnt models have been applied to Spanish more lately.…”

Section: Automatic Pos Tagging In Spanishmentioning

confidence: 96%

Etiquetado gramatical por entropía máxima y rasgos de similitud distribucional en un corpus subregional del español

2017

View full text Add to dashboard Cite

The present research study has used two state-of-the-art Spanish taggers with the primary goal of automatically tagging for POS a strictly assembled collection of unstructured text aimed at assisting a number of linguistic tasks, the subregional Mexican Corpus del Habla de Baja California (CHBC). These taggers, a Maximum-Entropy-based one and another one that adds to this statistical construct distributional similarity features, have recently been released but were missing an accuracy rate. Therefore, the second goal of this article is to evaluate and provide attested accuracy figures for the language models behind these taggers. In order to achieve these two goals, this article has proposed a novel, reduced tag set, which has also been proven useful for the goals here pursued. On a sample of almost 11,000 words and more than 12,500 tags for two genres (written text and transcribed oral speech), the Maximum Entropy tagger and the tagger with Maximum Entropy plus distributional similarity features have achieved results of 97.2% and 97.4%, respectively. By comparing these figures to a human ceiling or gold standard of 97.1%, also attested here, it is clear that the results of both taggers are competitive even when applied to an external data collection for which they have not been previously trained or tuned for. This is particularly important because under these kinds of experimental conditions taggers performance has been shown to deteriorate. Keywords: Mexican Spanish, stochastic POS tagging, tagged corpus. ResumenCon el objetivo primario de etiquetar automáticamente las categorías gramaticales en una colección de texto no estructurado, la cual fue diseñada para asistir en una serie de tareas lingüísticas, esta investigación ha utilizado dos etiquetadores automáticos de primera generación para el español. Estos etiquetadores han sido aplicados al Corpus del Habla de Baja California (CHBC) que cubre una subregión de México. Los dos etiquetadores, uno basado en el principio de Máxima Entropía y el otro que le suma a este modelo estadístico rasgos de similitud distribucional, son de reciente introducción y no se ha ofrecido un rango de precisión para los mismos. Por tanto, este artículo ha tenido como segundo objetivo el evaluar y proveer una cifra de precisión comprobada para los modelos de lenguaje que subyacen a los etiquetadores en cuestión. Con la finalidad de lograr estos dos objetivos, este artículo ha propuesto un etiquetario reducido, el cual también ha resultado de utilidad en la búsqueda de estos objetivos. Aplicados a una muestra de alrededor de 11,000 palabras y más de 12,500 etiquetas gramaticales para dos géneros (texto escrito y discurso oral transcrito), los dos etiquetadores, el de Máxima Entropía y el que suma a ésta los rasgos de similitud distribucional, han obtenido resultados de 97.2% y 97.4%, respectivamente. Al comparar estas cifras con el criterio estándar de 97.1% obtenido entre anotadores humanos, los resultados de ambos etiquetadores se muestran competitivos, incluso al aplicar...

show abstract

Section: Automatic Pos Tagging In Spanishmentioning

confidence: 96%

Etiquetado gramatical por entropía máxima y rasgos de similitud distribucional en un corpus subregional del español

2017

View full text Add to dashboard Cite

show abstract

“…POS tagging is widely adopted for languages such as English, German, Spanish and Arabic [1]- [4]. It plays a significant role in text analysis as it is an initial step to identify the grammar information in the text.…”

Section: Related Workmentioning

confidence: 99%

Benchmarking Mi-POS: Malay Part-of-Speech Tagger

Xian¹,

Lubani²,

Ping³

et al. 2016

IJKE

View full text Add to dashboard Cite

show abstract

“…Here, we will use CM to imply both. Work on computa- * This work was done during authors' internship at Microsoft Research India. tional models of CM have been few and far between (Solorio and Liu, 2008a;Solorio and Liu, 2008b;Nguyen and Dogruoz, 2013), primarily due to the paucity of CM data in conventional text-corpora which makes data-intensive methods hard to apply. Solorio and Liu (2008a) in their work on English-Spanish CM use models built on smaller datasets to predict valid switching points to synthetically generate data from monolingual corpora, and in another work (2008b) describe parts-of-speech (POS) tagging of CM text.…”

Section: Introductionmentioning

confidence: 99%

POS Tagging of English-Hindi Code-Mixed Social Media Content

Vyas¹,

Gella²,

Sharma³

et al. 2014

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)

144

View full text Add to dashboard Cite

Code-mixing is frequently observed in user generated content on social media, especially from multilingual users. The linguistic complexity of such content is compounded by presence of spelling variations, transliteration and non-adherance to formal grammar. We describe our initial efforts to create a multi-level annotated corpus of Hindi-English codemixed text collated from Facebook forums, and explore language identification, back-transliteration, normalization and POS tagging of this data. Our results show that language identification and transliteration for Hindi are two major challenges that impact POS tagging accuracy.

show abstract

Part-of-speech tagging for English-Spanish code-switched text

Cited by 94 publications

References 27 publications

Etiquetado gramatical por entropía máxima y rasgos de similitud distribucional en un corpus subregional del español

Etiquetado gramatical por entropía máxima y rasgos de similitud distribucional en un corpus subregional del español

Benchmarking Mi-POS: Malay Part-of-Speech Tagger

POS Tagging of English-Hindi Code-Mixed Social Media Content

Contact Info

Product

Resources

About