MultiFiT: Efficient Multi-lingual Language Model Fine-tuning

Eisenschlos, Julian Martin; Ruder, Sebastian; Czapla, Piotr; Kardas, Marcin; Gugger, Sylvain; Howard, Jeremy

doi:10.48550/arxiv.1909.04761

Cited by 2 publications

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…-USE + SVM: We first adopt a ML-based approach. Instead of TF-IDF features, we obtain contextualized representations of the input using Universal sentence encoder (USE) 6 . We then fed the input representations to an SVM model.…”

Section: Comparison With Other Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Fake News Detection System using XLNet model with Topic Distributions: CONSTRAINT@AAAI2021 Shared Task

Gautam¹,

Venktesh²,

Masud³

2021

Preprint

View full text Add to dashboard Cite

With the ease of access to information, and its rapid dissemination over the internet (both velocity and volume), it has become challenging to filter out truthful information from fake ones. The research community is now faced with the task of automatic detection of fake news, which carries real-world socio-political impact. One such research contribution came in the form of the Constraint@AAA12021 Shared Task on COVID19 Fake News Detection in English. In this paper, we shed light on a novel method we proposed as a part of this shared task. Our team introduced an approach to combine topical distributions from Latent Dirichlet Allocation (LDA) with contextualized representations from XLNet. We also compared our method with existing baselines to show that XLNet + Topic Distributions outperforms other approaches by attaining an F1-score of 0.967.

show abstract

Section: Comparison With Other Methodsmentioning

confidence: 99%

“…Recently, rather than leveraging hand-crafted features, automatic extraction of relevant features in the form of distributed representations has become popular [21]. Various previous studies [7,6,20] have shown the effect usage of Language Model Fine-tuning are an better alternative for the classification tasks than other methods.…”

Section: Introductionmentioning

confidence: 99%

Fake News Detection System using XLNet model with Topic Distributions: CONSTRAINT@AAAI2021 Shared Task

Gautam¹,

Venktesh²,

Masud³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Strategic knowledge sharing has been shown to improve the performance on downstream tasks and languages (Gururangan et al, 2020). Therefore, this technique is crucial for multilingual applications, as most of the world's languages lack large amount of labeled data Eisenschlos et al, 2019;Joshi et al, 2020). However the performance of multilingual language model on low-resource languages is still limited compared to other languages since they are naturally under-sampled during the training process (Wu and Dredze, 2020).…”

Section: Transfer Learning For Low-resource Languagesmentioning

confidence: 99%

Can Character-based Language Models Improve Downstream Task Performance in Low-Resource and Noisy Language Scenarios?

Riabi¹,

Sagot²,

Seddah³

2021

Preprint

View full text Add to dashboard Cite

Recent impressive improvements in NLP, largely based on the success of contextual neural language models, have been mostly demonstrated on at most a couple dozen highresource languages. Building language models and, more generally, NLP systems for nonstandardized and low-resource languages remains a challenging task. In this work, we focus on North-African colloquial dialectal Arabic written using an extension of the Latin script, called NArabizi, found mostly on social media and messaging communication. In this low-resource scenario with data displaying a high level of variability, we compare the downstream performance of a character-based language model on part-of-speech tagging and dependency parsing to that of monolingual and multilingual models. We show that a characterbased model trained on only 99k sentences of NArabizi and fined-tuned on a small treebank of this language leads to performance close to those obtained with the same architecture pretrained on large multilingual and monolingual models. Confirming these results a on much larger data set of noisy French user-generated content, we argue that such character-based language models can be an asset for NLP in low-resource and high language variability settings.

show abstract

MultiFiT: Efficient Multi-lingual Language Model Fine-tuning

Cited by 2 publications

References 0 publications

Fake News Detection System using XLNet model with Topic Distributions: CONSTRAINT@AAAI2021 Shared Task

Fake News Detection System using XLNet model with Topic Distributions: CONSTRAINT@AAAI2021 Shared Task

Can Character-based Language Models Improve Downstream Task Performance in Low-Resource and Noisy Language Scenarios?

Contact Info

Product

Resources

About