Towards a Universal Sentiment Classifier in Multiple languages

Modern NLP applications have enjoyed a great boost utilizing neural networks models. Such deep neural models, however, are not applicable to most human languages due to the lack of annotated training data for various NLP tasks. Cross-lingual transfer learning (CLTL) is a viable method for building NLP models for a low-resource target language by leveraging labeled data from other (source) languages. In this work, we focus on the multilingual transfer setting where training data in multiple source languages is leveraged to further boost target language performance.Unlike most existing methods that rely only on language-invariant features for CLTL, our approach coherently utilizes both languageinvariant and language-specific features at instance level. Our model leverages adversarial networks to learn language-invariant features, and mixture-of-experts models to dynamically exploit the similarity between the target language and each individual source language 1 . This enables our model to learn effectively what to share between various languages in the multilingual setup. Moreover, when coupled with unsupervised multilingual embeddings, our model can operate in a zero-resource setting where neither target language training data nor cross-lingual resources are available. Our model achieves significant performance gains over prior art, as shown in an extensive set of experiments over multiple text classification and sequence tagging tasks including a large-scale industry dataset. Shared Feature Extractor Fs MoE Private Feature Extractor Fp MoE Task-Specific Predictor C Task Label Multilingual Word Representation Input Text J C Language Discriminator D Language Label J D 1 J D Gate Label 2 J g Forward and backward passes when updating the parameters of Fs, Fp and C Forward and backward passes when updating the parameters of D

show abstract

“…Prettenhofer and Stein (2010) 2 Xiao and Guo (2013) 3Pham et al (2015) 4 Xu and Wan (2017) 5Xu and Yang (2017) …”

mentioning

confidence: 99%

Multi-Source Cross-Lingual Model Transfer: Learning What to Share

Chen

Awadallah

Hassan

et al. 2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…For this extended abstract, we have updated the list of baseline methods by adding new approaches (TCT [Huang et al, 2017], TrAdaB [Huang et al, 2017, DANN [Ganin et al, 2016], CL-TS [Zhou et al, 2015], Bi-PV [Xu and Wan, 2017], BiDRL [Zhou et al, 2016b], WSDNNs, [Zhou et al, 2016a], CLDFA [Xu and Yang, 2017]) which have been published in the cross-domain and cross-lingual arena after our original work [Moreo Fernández et al, 2016], and kept those which performed best in our original evaluation (SCL-MI [Blitzer et al, 2007], SFA [Pan et al, 2010, SDA [Glorot et al, 2011], and SSMC [Xiao and Guo, 2014]). We also consider an upper bound that trains the classifier on the training set of the target domain ("Upper"), and a lower bound that trains the classifier on the source domain and then applies the trained classifier directly in the target domain, i.e., without carrying out any sort of knowledge transfer ("No-Trans").…”

Section: Methodsmentioning

confidence: 99%

Distributional Correspondence Indexing for Cross-Lingual and Cross-Domain Sentiment Classification (Extended Abstract)

Moreo

Esuli

Sebastiani

2018

Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence

View full text Add to dashboard Cite

Domain Adaptation (DA) techniques aim at enabling machine learning methods learn effective classifiers for a “target” domain when the only available training data belongs to a different “source” domain. In this extended abstract, we briefly describe our new DA method called Distributional Correspondence Indexing (DCI) for sentiment classification. DCI derives term representations in a vector space common to both domains where each dimension reflects its distributional correspondence to a pivot, i.e., to a highly predictive term that behaves similarly across domains. The experiments we have conducted show that DCI obtains better performance than current state-of-the-art techniques for cross-lingual and cross-domain sentiment classification.

show abstract

“…However, single-source CLT methods would incur the risk of negative transfer when there exists a large language shift. Alternately, multi-source CLT (McDonald et al, 2011;Xu and Wan, 2017;, transferring from multiple source languages, has been proved to increase the stability of the transfer. Another research efforts made on cross-lingual word representation learning (Zou et al, 2013;Mikolov et al, 2013;Conneau et al, 2018a;Artetxe et al, 2018) and mPLM (Devlin et al, 2019;Lample and Conneau, 2019;Eisenschlos et al, 2019;Chidambaram et al, 2019), which exploit unsupervised learning on large-scale multilingual corpus to learn versatile multilingual contextualized embeddings.…”

Section: Cross-lingual Transfermentioning

confidence: 99%

Learn to Cross-lingual Transfer with Meta Graph Learning Across Heterogeneous Languages

Kumar

Headden

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

The recent emergence of multilingual pretraining language model (mPLM) has enabled breakthroughs on various downstream crosslingual transfer (CLT) tasks. However, mPLMbased methods usually involve two problems:(1) simply fine-tuning may not adapt generalpurpose multilingual representations to be task-aware on low-resource languages; (2) ignore how cross-lingual adaptation happens for downstream tasks. To address the issues, we propose a meta graph learning (MGL) method. Unlike prior works that transfer from scratch, MGL can learn to cross-lingual transfer by extracting meta-knowledge from historical CLT experiences (tasks), making mPLM insensitive to low-resource languages. Besides, for each CLT task, MGL formulates its transfer process as information propagation over a dynamic graph, where the geometric structure can automatically capture intrinsic language relationships to guide cross-lingual transfer explicitly. Empirically, extensive experiments on both public and real-world datasets demonstrate the effectiveness of the MGL method.

show abstract

Towards a Universal Sentiment Classifier in Multiple languages

Cited by 17 publications

References 28 publications

Multi-Source Cross-Lingual Model Transfer: Learning What to Share

Multi-Source Cross-Lingual Model Transfer: Learning What to Share

Distributional Correspondence Indexing for Cross-Lingual and Cross-Domain Sentiment Classification (Extended Abstract)

Learn to Cross-lingual Transfer with Meta Graph Learning Across Heterogeneous Languages

Contact Info

Product

Resources

About