Language-agnostic BERT Sentence Embedding

Feng, Fangxiaoyu; Yang, Yan; Cer, Daniel; Arivazhagan, Naveen; Wang, Wei

doi:10.48550/arxiv.2007.01852

Cited by 108 publications

(177 citation statements)

References 15 publications

Supporting

Mentioning

150

Contrasting

Unclassified

Order By: Relevance

“…The same trends in the results were observed with other standard multilingual sentence encoders such as LaBSE(Feng et al, 2020), see Appendix C for additional results.…”

supporting

confidence: 80%

Cross-Lingual Dialogue Dataset Creation via Outline-Based Generation

Majewska¹,

Razumovskaia²,

Ponti³

et al. 2022

Preprint

View full text Add to dashboard Cite

Multilingual task-oriented dialogue (TOD) facilitates access to services and information for many (communities of) speakers. Nevertheless, the potential of this technology is not fully realised, as current datasets for multilingual TOD-both for modular and end-toend modelling-suffer from severe limitations. 1) When created from scratch, they are usually small in scale and fail to cover many possible dialogue flows. 2) Translation-based TOD datasets might lack naturalness and cultural specificity in the target language. In this work, to tackle these limitations we propose a novel outline-based annotation process for multilingual TOD datasets, where domainspecific abstract schemata of dialogue are mapped into natural language outlines. These in turn guide the target language annotators in writing a dialogue by providing instructions about each turn's intents and slots. Through this process we annotate a new large-scale dataset for training and evaluation of multilingual and cross-lingual TOD systems. Our Cross-lingual Outline-based Dialogue dataset (termed COD) enables natural language understanding, dialogue state tracking, and end-toend dialogue modelling and evaluation in 4 diverse languages: Arabic, Indonesian, Russian, and Kiswahili. Qualitative and quantitative analyses of COD versus an equivalent translation-based dataset demonstrate improvements in data quality, unlocked by the outline-based approach. Finally, we benchmark a series of state-of-the-art systems for cross-lingual TOD, setting reference scores for future work and demonstrating that COD prevents over-inflated performance, typically met with prior translation-based TOD datasets.

show abstract

“…The same trends in the results were observed with other standard multilingual sentence encoders such as LaBSE(Feng et al, 2020), see Appendix C for additional results.…”

supporting

confidence: 80%

Cross-Lingual Dialogue Dataset Creation via Outline-Based Generation

Majewska¹,

Razumovskaia²,

Ponti³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Besides the multilingual version of the classic BERT, there is also a Languageagnostic BERT Sentence Embedding (LaBSE) [19]. It is a multilingual version of BERT with sentence embeddings similar to Sentence-BERT [36].…”

Section: Transformers and Bertmentioning

confidence: 99%

Towards Trustworthy AutoGrading of Short, Multi-lingual, Multi-type Answers

Schneider¹,

Richner²,

Riser³

2022

Preprint

View full text Add to dashboard Cite

Autograding short textual answers has become much more feasible due to the rise of NLP and the increased availability of question-answer pairs brought about by a shift to online education. Autograding performance is still inferior to human grading. The statistical and black-box nature of state-of-the-art machine learning models makes them untrustworthy, raising ethical concerns and limiting their practical utility. Furthermore, the evaluation of autograding is typically confined to small, monolingual datasets for a specific question type. This study uses a large dataset consisting of about 10 million question-answer pairs from multiple languages covering diverse fields such as math and language, and strong variation in question and answer syntax. We demonstrate the effectiveness of finetuning transformer models for autograding for such complex datasets. Our best hyperparameter-tuned model yields an accuracy of about 86.5%, comparable to the state-of-the-art models that are less general and more tuned to a specific type of question, subject, and language. More importantly, we address trust and ethical concerns. By involving humans in the autograding process, we show how to improve the accuracy of automatically graded answers, achieving accuracy equivalent to that of teaching assistants. We also show how teachers can effectively control the type of errors made by the system and how they can validate efficiently that the autograder's performance on individual exams is close to the expected performance.

show abstract

“…• LaBSE [13], a language-agnostic BERT sentence embedding model pre-trained on texts in 109 languages.…”

Section: Modelsmentioning

confidence: 99%

Fine-tuning of Pre-trained Transformers for Hate, Offensive, and Profane Content Detection in English and Marathi

Glazkova¹,

Michael²,

Glazkov³

2021

Preprint

View full text Add to dashboard Cite

This paper describes neural models developed for the Hate Speech and Offensive Content Identification in English and Indo-Aryan Languages Shared Task 2021. Our team called neuro-utmn-thales participated in two tasks on binary and fine-grained classification of English tweets that contain hate, offensive, and profane content (English Subtasks A & B) and one task on identification of problematic content in Marathi (Marathi Subtask A). For English subtasks, we investigate the impact of additional corpora for hate speech detection to fine-tune transformer models. We also apply a one-vs-rest approach based on Twitter-RoBERTa to discrimination between hate, profane and offensive posts. Our models ranked third in English Subtask A with the F1-score of 81.99% and ranked second in English Subtask B with the F1-score of 65.77%. For the Marathi tasks, we propose a system based on the Language-Agnostic BERT Sentence Embedding (LaBSE). This model achieved the second result in Marathi Subtask A obtaining an F1 of 88.08%.

show abstract

Language-agnostic BERT Sentence Embedding

Cited by 108 publications

References 15 publications

Cross-Lingual Dialogue Dataset Creation via Outline-Based Generation

Cross-Lingual Dialogue Dataset Creation via Outline-Based Generation

Towards Trustworthy AutoGrading of Short, Multi-lingual, Multi-type Answers

Fine-tuning of Pre-trained Transformers for Hate, Offensive, and Profane Content Detection in English and Marathi

Contact Info

Product

Resources

About