Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling

Han, Xiaochuang; Eisenstein, Jacob

doi:10.18653/v1/d19-1433

Cited by 143 publications

(152 citation statements)

References 33 publications

(38 reference statements)

Supporting

Mentioning

137

Contrasting

Order By: Relevance

“…We also experimented with searching concepts in SNOMED-CT using the Meaning Cloud tool 12 , however it did not work well, as many concepts for the shared task were annotated based on their synonyms.…”

Section: Number Of Training Epochsmentioning

confidence: 99%

See 1 more Smart Citation

Testing Contextualized Word Embeddings to Improve NER in Spanish Clinical Case Narratives

et al. 2020

View full text Add to dashboard Cite

In the Big Data era, there is an increasing need to fully exploit and analyze the huge quantity of information available about health. Natural Language Processing (NLP) technologies can contribute by extracting relevant information from unstructured data contained in Electronic Health Records (EHR) such as clinical notes, patients' discharge summaries and radiology reports. The extracted information can help in health-related decision making processes. The Named Entity Recognition (NER) task, which detects important concepts in texts (e.g., diseases, symptoms, drugs, etc.), is crucial in the information extraction process yet has received little attention in languages other than English. In this work, we develop a deep learning-based NLP pipeline for biomedical entity extraction in Spanish clinical narratives. We explore the use of contextualized word embeddings, which incorporate context variation into word representations, to enhance named entity recognition in Spanish language clinical text, particularly of pharmacological substances, compounds, and proteins. Various combinations of word and sense embeddings were tested on the evaluation corpus of the PharmacoNER 2019 task, the Spanish Clinical Case Corpus (SPACCC). This data set consists of clinical case sections extracted from open access Spanishlanguage medical publications. Our study shows that our deep-learning-based system with domain-specific contextualized embeddings coupled with stacking of complementary embeddings yields superior performance over a system with integrated standard and general-domain word embeddings. With this system, we achieve performance competitive with the state-of-the-art. INDEX TERMS Clinical case narratives, Contextualized word embeddings, Deep learning, Language representations, Named entity recognition, Natural language processing, Spanish language Recent developments in NLP have shown the advantage

show abstract

Section: Number Of Training Epochsmentioning

confidence: 99%

“…The embedding function is trained either from a language modeling perspective [10] or based on recovering masked parts of tokens [11]. The downstream tasks which incorporate these embeddings are considered to be learned in a semi-supervised manner because they benefit from large amounts of unlabeled data [12], [13].…”

Section: Introductionmentioning

confidence: 99%

Testing Contextualized Word Embeddings to Improve NER in Spanish Clinical Case Narratives

et al. 2020

View full text Add to dashboard Cite

show abstract

“…One possible direction to improve pre-trained multilingual LMs for MRC questions in a target language is to apply unsupervised domain adaptation [44]. Hundreds of questions would not be enough for a multilingual LM to fully adapt to both the target language and the downstream task.…”

Section: Modelmentioning

confidence: 99%

Assessment of Word-Level Neural Language Models for Sentence Completion

Park

2020

Applied Sciences

View full text Add to dashboard Cite

The task of sentence completion, which aims to infer the missing text of a given sentence, was carried out to assess the reading comprehension level of machines as well as humans. In this work, we conducted a comprehensive study of various approaches for the sentence completion based on neural language models, which have been advanced in recent years. First, we revisited the recurrent neural network language model (RNN LM), achieving highly competitive results with an appropriate network structure and hyper-parameters. This paper presents a bidirectional version of RNN LM, which surpassed the previous best results on Microsoft Research (MSR) Sentence Completion Challenge and the Scholastic Aptitude Test (SAT) sentence completion questions. In parallel with directly applying RNN LM to sentence completion, we also employed a supervised learning framework that fine-tunes a large pre-trained transformer-based LM with a few sentence-completion examples. By fine-tuning a pre-trained BERT model, this work established state-of-the-art results on the MSR and SAT sets. Furthermore, we performed similar experimentation on newly collected cloze-style questions in the Korean language. The experimental results reveal that simply applying the multilingual BERT models for the Korean dataset was not satisfactory, which leaves room for further research.results with classical non-neural feature based methods. In [9], the authors introduced a neural model named context2vec, which embeds a target word by considering the surrounding sentential context, demonstrating its usefulness in sentence completion in addition to word sense disambiguation and lexical substitution. Tran et al. [10] established the state-of-the-art results on the MSR set with Recurrent Memory Network (RMN), which stacked memory network blocks on RNN for language modeling.Recently, Park et al. [11] revisited the word-level RNN LM based approach for sentence completion. Motivated by the empirical fact that the performance of the RNN LM highly depends on the number of nodes and optimization parameters [12,13], Park et al. demonstrated that their implementation of RNN LM surpassed the state-of-the-art models on the MSR set despite its simple architecture. Furthermore, they proposed a bidirectional version, which delivered additional performance gains by exploiting future context information. The authors also validated the RNN LMs against the SAT dataset, and they achieved higher accuracy than the other previously published results.This work extends the study of Park et al. [11] with extensive experiments on various sentence completion methods based on neural LMs. To clarify which modification of the RNN LM mainly brings the performance gain, we added more experimental results for different choices of the network. Furthermore, this paper introduces and compares three criteria for selecting the answer based on a trained LM for sentence completion.This study also includes a supervised learning approach that directly receives supervision from sentence completion questions....

show abstract

“…However, the target domain usually differs from the pre-training corpus which may result in the unsatisfactory performance of the model on the downstream task. Recently, unsupervised domain adaptation of language models has shown quality improvement in a number of NLP tasks, including sequence labelling [9]. [10] study unsupervised domain adaptation of BERT in the limited labelled and unlabelled data scenarios.…”

Section: Related Workmentioning

confidence: 99%

Domain-Transferable Method for Named Entity Recognition Task

Mikhailov¹,

Shavrina²

2020

Computer Science &Amp; Information Technology (CS &Amp; IT)

View full text Add to dashboard Cite

Named Entity Recognition (NER) is a fundamental task in the fields of natural language processing and information extraction. NER has been widely used as a standalone tool or an essential component in a variety of applications such as question answering, dialogue assistants and knowledge graphs development. However, training reliable NER models requires a large amount of labelled data which is expensive to obtain, particularly in specialized domains. This paper describes a method to learn a domain-specific NER model for an arbitrary set of named entities when domain-specific supervision is not available. We assume that the supervision can be obtained with no human effort, and neural models can learn from each other. The code, data and models are publicly available.

show abstract

Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling

Cited by 143 publications

References 33 publications

Testing Contextualized Word Embeddings to Improve NER in Spanish Clinical Case Narratives

Testing Contextualized Word Embeddings to Improve NER in Spanish Clinical Case Narratives

Assessment of Word-Level Neural Language Models for Sentence Completion

Domain-Transferable Method for Named Entity Recognition Task

Contact Info

Product

Resources

About