Semi-Supervised Learning on Meta Structure: Multi-Task Tagging and Parsing in Low-Resource Scenarios

Abstract

Multi-view learning makes use of diverse models arising from multiple sources of input or different feature subsets for the same task. For example, a given natural language processing task can combine evidence from models arising from character, morpheme, lexical, or phrasal views. The most common strategy with multi-view learning, especially popular in the neural network community, is to unify multiple representations into one unified vector through concatenation, averaging, or pooling, and then build a single-view model on top of the unified representation. As an alternative, we examine whether building one model per view and then unifying the different models can lead to improvements, especially in low-resource scenarios. More specifically, taking inspiration from co-training methods, we propose a semi-supervised learning approach based on multi-view models through consensus promotion, and investigate whether this improves overall performance. To test the multi-view hypothesis, we use moderately low-resource scenarios for nine languages and test the performance of the joint model for part-of-speech tagging and dependency parsing. The proposed model shows significant improvements across the test cases, with average gains of -0.9 ∼ +9.3 labeled attachment score (LAS) points. We also investigate the effect of unlabeled data on the proposed model by varying the amount of training data and by using different domains of unlabeled data.

Neural Unsupervised Domain Adaptation in NLP—A Survey

“…They include models trained on other grammar formalisms to improve dependency parsing on Twitter (Foster et al, 2011). Recently, this line of classics has been revisited (Ruder and Plank, 2018;Rotman and Reichart, 2019;Lim et al, 2020). For example, classic methods such as tri-training constitute a strong baseline for domain shift in neural times (Ruder and Plank, 2018).…”

Part-of-Speech Tagging Using Multiview Learning

“…Compared with taggers that used BERT-like models, our join model shows slightly better performances. Even though Co-meta applied both the meta-LSTM and the sentence-based character model [15], our join model that applies two character models showed higher performances. However, it should be noted that the udfy model was first trained with 75 different languages and then tuned for English [11].…”

“…In POS tagging, multilingual BERT 6 that can handle 100 languages is applied to train a multilingual POS tagger, namely, udify [11] in 2019. More recently, Lim et al [15] proposed a Co-meta tagger that leverages its performance based on a semi-supervised learning approach with the multilingual BERT and BERT-base monolingual English model, and they achieved SOTA results. We compare the performance of our tagger with BERT-like models as existing SOTA systems, in particular, with multilingual BERT, BERT-base, and Roberta [35].…”

“…From a technical point of view, ELMo can be used as a function that provides a deep contextualized word representation based on the entire input sentence using a pretrained LSTM. Recently, BERT [10] trained by bidirectional transformers with a masked language model strategy showed good results in tagging and parsing [11], [15], [27]. As an external word view (representation), we use both ELMo and BERT representations on our tagger.…”

“…To address this issue, one of the most promising approaches is to adapt meta-LSTMs on top of several subword models. Following on the previously proposed joint methods that promote the multiview approach with meta-LSTMs [14], [15], we propose a joint learning method for the better application of several subword models. This method can be trained using two novel lexical representations:…”