ELMo and BERT in semantic change detection for Russian

Rodina, Julia; Trofimova, Yuliya; Kutuzov, Andrey; Artemova, Ekaterina

doi:10.48550/arxiv.2010.03481

Cited by 1 publication

(2 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Within contextual LMs, BERT and ELMo per-form similarly for Russian. Rodina et al 2020 show that neither BERT nor Elmo outperform each other when fine-tuned using historical text in Russian to detect semantic change. Moreover, results from the shared task on Unsupervised Lexical Semantic Change Detection in 5 languages hosted at SemEval 2020 show that systems performing well over one language may not perform as well for other languages.…”

Section: Time Periodmentioning

confidence: 87%

See 1 more Smart Citation

On the Universality of Deep Contextual Language Models

Bhatt¹,

Goyal²,

Dandapat³

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep Contextual Language Models (LMs) like ELMO, BERT, and their successors dominate the landscape of Natural Language Processing due to their ability to scale across multiple tasks rapidly by pre-training a single model, followed by task-specific fine-tuning. Furthermore, multilingual versions of such models like XLM-R and mBERT have given promising results in zero-shot cross-lingual transfer, potentially enabling NLP applications in many under-served and under-resourced languages. Due to this initial success, pre-trained models are being used as 'Universal Language Models' as the starting point across diverse tasks, domains, and languages. This work explores the notion of 'Universality' by identifying seven dimensions across which a universal model should be able to scale, that is, perform equally well or reasonably well, to be useful across diverse settings. We outline the current theoretical and empirical results that support model performance across these dimensions, along with extensions that may help address some of their current limitations. Through this survey, we lay the foundation for understanding the capabilities and limitations of massive contextual language models and help discern research gaps and directions for future work to make these LMs inclusive and fair to diverse applications, users, and linguistic phenomena.1 Throughout the rest of the paper -"these models", "LMs", "general domain LMs", "contextual LMs", "universal LMs" and all such terms refers to models including but not limited to ELMo, BERT, RoBERTa, GPT their variants, successors and multilingual versions

show abstract

Section: Time Periodmentioning

confidence: 87%

“…While the success in identifying these shifts may be limited, (Rodina et al, 2020) find that DSC identified by contextual LMs can have a strong correlation with human judgment of change.…”

Section: Time Periodmentioning

confidence: 93%