ETNLP: A Visual-Aided Systematic Approach to Select Pre-Trained Embeddings for a Downstream Task

Abstract

Given many recent advanced embedding models, selecting pre-trained word embedding (a.k.a., word representation) models best fit for a specific downstream task is non-trivial. In this paper, we propose a systematic approach, called ETNLP, for extracting, evaluating, and visualizing multiple sets of pre-trained word embeddings to determine which embeddings should be used in a downstream task.We demonstrate the effectiveness of the proposed approach on our pre-trained word embedding models in Vietnamese to select which models are suitable for a named entity recognition (NER) task. Specifically, we create a large Vietnamese word analogy list to evaluate and select the pre-trained embedding models for the task. We then utilize the selected embeddings for the NER task and achieve the new state-of-the-art results on the task benchmark dataset. We also apply the approach to another downstream task of privacy-guaranteed embedding selection, and show that it helps users quickly select the most suitable embeddings. In addition, we create an open-source system using the proposed systematic approach to facilitate similar studies on other NLP tasks. The source code and data are available at https: //github.com/vietnlp/etnlp.

PhoBERT: Pre-trained language models for Vietnamese

“…For NER, PhoBERT large produces 1.1 points higher F 1 than PhoBERT base . In addition, PhoBERT base obtains 2+ points higher than the previous SOTA feature-and neural network-based models VnCoreNLP-NER and BiLSTM-CNN-CRF (Ma and Hovy, 2016) [ ] 88.6 BiLSTM-max (Conneau et al, 2018) 66.4 VNER (Nguyen et al, 2019b) 89.6 mBiLSTM (Artetxe and Schwenk, 2019) 72.0 BiLSTM-CNN-CRF + ETNLP [♠] 91.1 multilingual BERT (Devlin et al, 2019) [ ] 69.5 VnCoreNLP-NER + ETNLP [♠] 91.3 XLM MLM+TLM (Conneau and Lample, 2019) 76.6 XLM-R base (our result) 92.0 XLM-R base (Conneau et al, 2020) 75.4 XLM-R large (our result) 92.8 XLM-R large (Conneau et al, 2020) 79.7 PhoBERT base 93.6 PhoBERT base 78.5 PhoBERT large 94.7 PhoBERT large 80.0 are trained with the set of 15K BERT-based ETNLP word embeddings (Vu et al, 2019).…”

“…The success of pre-trained BERT and its variants has largely been limited to the English language. For other languages, one could retrain a language-specific model using the BERT architecture (Cui et al, 2019;de Vries et al, 2019;Vu et al, 2019;Martin et al, 2020) or employ existing pre-trained multilingual BERT-based models (Devlin et al, 2019;Conneau and Lample, 2019;Conneau et al, 2020).…”

“…• The Vietnamese Wikipedia corpus is the only data used to train monolingual language models (Vu et al, 2019), and it also is the only Vietnamese dataset which is included in the pretraining data used by all multilingual language * Work done during internship at VinAI Research. models except XLM-R.…”

“…Thang et al (2008) show that 85% of Vietnamese word types are composed of at least two syllables.2 Although performing word segmentation before applying BPE on the Vietnamese Wikipedia corpus, ETNLP(Vu et al, 2019) in fact does not publicly release any pre-trained BERT-based language model (https://github.com/ vietnlp/etnlp). In particular,Vu et al (2019) release a set of 15K BERT-based word embeddings specialized only for the Vietnamese NER task.…”

Enhancing Lexical-Based Approach With External Knowledge for Vietnamese Multiple-Choice Machine Reading Comprehension

“…In this model, we use external knowledge sources as word embeddings. To explore the effectiveness of word embeddings, we evaluate the performance of our proposed model on with several word embeddings including Word2vec [73], Word2vec and Character2vec [74], fastText [75], ELMo [72], BERT [49] and MULTI [76]. In particular, we use pre-trained embeddings on Vietnamese Wikipedia proposed by [76] for all experiments of our proposed method.…”