It’s not Greek to mBERT: Inducing Word-Level Translations from Multilingual BERT

Gonen, Hila; Ravfogel, Shauli; Elazar, Yanai; Goldberg, Yoav

doi:10.18653/v1/2020.blackboxnlp-1.5

Cited by 21 publications

(17 citation statements)

References 16 publications

(22 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other work has taken this further by focusing on the hypothesis that mBERT encodings contain both a language-specific and a language-neutral component (Libovický et al, 2020). Gonen et al (2020) set out to disentangle both components and find that in 'language identity subspace', t-SNE projections show large improvement in clustering with respect to language. In language-neutral space, semantic representations are largely intact.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

On the Language-specificity of Multilingual BERT and the Impact of Fine-tuning

Tanti¹,

Plas²,

Borg³

et al. 2021

Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP

View full text Add to dashboard Cite

Recent work has shown evidence that the knowledge acquired by multilingual BERT (mBERT) has two components: a languagespecific and a language-neutral one. This paper analyses the relationship between them, in the context of fine-tuning on two tasks -POS tagging and natural language inferencewhich require the model to bring to bear different degrees of language-specific knowledge. Visualisations reveal that mBERT loses the ability to cluster representations by language after fine-tuning, a result that is supported by evidence from language identification experiments. However, further experiments on 'unlearning' language-specific representations using gradient reversal and iterative adversarial learning are shown not to add further improvement to the language-independent component over and above the effect of fine-tuning. The results presented here suggest that the process of fine-tuning causes a reorganisation of the model's limited representational capacity, enhancing language-independent representations at the expense of language-specific ones.

show abstract

Section: Related Workmentioning

confidence: 99%

“…These results have motivated researchers to try and disentangle the language-specific and languageneutral components of mBERT (e.g. Libovický et al, 2020;Gonen et al, 2020).…”

Section: Introductionmentioning

confidence: 99%

On the Language-specificity of Multilingual BERT and the Impact of Fine-tuning

Tanti¹,

Plas²,

Borg³

et al. 2021

Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP

View full text Add to dashboard Cite

show abstract

“…mBERT has been implemented for languages like Bangla, Greek, Danish, Turkish, etc. It contributes to multilingual text classification [25] , [26] , [33] , offensive language detection [18] , Word Sense Disambiguation [53] , Translation Quality Estimation [16] , [22] , etc.…”

Section: Introductionmentioning

confidence: 99%

Reading comprehension based question answering system in Bangla language with transformer-based learning

Aurpa

Rifat

Ahmed³

et al. 2022

Heliyon

View full text Add to dashboard Cite

“…The XLM-RoBERTa is a transformer model created by the coders at Facebook in 2019 [8]. This model is superior to the previous multilingual bidirectional encoder representations from transformers (mBERT) model which is only a multilingual transformer models [9]. The mBERT model is an evolution from the early BERT model which is a transformer model that can identify the attention in a text, thus predicting the following words based on the attention from each word [10].…”

mentioning

confidence: 99%

Mengoptimalkan Akurasi pada Klasifikasi Emosi Majemuk Berdasarkan Semantik Kalimat Menggunakan XLM-RoBERTa

Aripin

Santoso²,

Haryanto³

2023

JNTETI

View full text Add to dashboard Cite

Emosi dasar dibagi menjadi enam, yaitu marah, sedih, senang, jijik, kaget, dan takut. Gabungan lebih dari satu emosi dasar dapat menciptakan sebuah emosi baru, yaitu emosi majemuk. Emosi majemuk dapat diimplementasikan untuk chat-bot, penerjemahan bahasa, text summarization, dan sebagainya. Penelitian mengenai klasifikasi emosi berdasarkan teks bahasa Indonesia telah banyak dilakukan dengan menggunakan beberapa model tradisional, seperti multinomial naïve Bayes, SVM, k-nearest neighborhood, dan term frequency–inverse document frequency (TF-IDF). Penelitian tersebut memiliki kelemahan, antara lain kinerja yang kurang optimal karena model hanya dapat mengklasifikasi dari data yang telah dipelajarinya, diperlukan pemrosesan teks terlebih dahulu, dan diperlukannya waktu yang lama dalam proses pelatihan dengan data berukuran besar. Penelitian ini bertujuan untuk mengatasi beberapa kelemahan penelitian sebelumnya dengan menggunakan model cross-lingual language model-robustly optimized bidirectional encoder representations from transformers approach (XML-RoBERTa) untuk mengklasifikasi emosi majemuk berdasarkan semantik atau makna kalimat dan kata. XLM-RoBERTa merupakan sebuah model transformer yang dapat mengetahui sebuah makna kata dari attention mechanism pada kata tersebut dan merupakan sebuah vektor yang merepresentasikan sebuah konteks atau makna kata. Attention mechanism merupakan sebuah representasi kata berbentuk vektor untuk mengetahui penggunaan dan posisi kata pada suatu kalimat dan merupakan cara agar model dapat mengetahui makna dari sebuah kata. Dengan attention mechanism, model dapat melihat pola kalimat dari penggunaan kata dan mengklasifikasikan kalimat tersebut sesuai dengan pola dan urutan kata, sehingga semantik kalimat dapat diketahui. Hasil eksperimen menunjukkan bahwa model yang diusulkan mampu mengklasifikasi teks berbahasa Indonesia ke dalam kelas-kelas emosi dasar dan kombinasinya sebagai dasar pembentukan emosi majemuk dengan akurasi sebesar 95,56%. Nilai akurasi ini merupakan nilai akurasi yang lebih unggul dibandingkan dengan penelitian klasifikasi kelas emosi majemuk dengan menggunakan model tradisional.

show abstract

It’s not Greek to mBERT: Inducing Word-Level Translations from Multilingual BERT

Cited by 21 publications

References 16 publications

On the Language-specificity of Multilingual BERT and the Impact of Fine-tuning

On the Language-specificity of Multilingual BERT and the Impact of Fine-tuning

Reading comprehension based question answering system in Bangla language with transformer-based learning

Mengoptimalkan Akurasi pada Klasifikasi Emosi Majemuk Berdasarkan Semantik Kalimat Menggunakan XLM-RoBERTa

Contact Info

Product

Resources

About