Word Sense Disambiguation using a Bidirectional LSTM

Abstract

In this paper we present a clean, yet effective, model for word sense disambiguation. Our approach leverage a bidirectional long short-term memory network which is shared between all words. This enables the model to share statistical strength and to scale well with vocabulary size. The model is trained end-to-end, directly from the raw text to sense labels, and makes effective use of word order. We evaluate our approach on two standard datasets, using identical hyperparameter settings, which are in turn tuned on a third set of held out data. We employ no external resources (e.g. knowledge graphs, part-of-speech tagging, etc), language specific features, or hand crafted rules, but still achieve statistically equivalent results to the best state-of-the-art systems, that employ no such limitations.

GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge

“…Bi-LSTM (Kågebäck and Salomonsson, 2016) is a baseline for neural models. Bi-LSTM +att.+LEX+P OS (Raganato et al, 2017a) is a multi-task learning framework for WSD, POS tagging, and LEX with self-attention mechanism, which converts WSD to a sequence learning task.…”

“…Recent neural-based methods are devoted to dealing with this problem. Kågebäck and Salomonsson (2016) present a supervised classifier based on Bi-LSTM, which shares parameters among all word types except the last layer. Raganato et al (2017a) convert WSD task to a sequence labeling task, thus building a unified model for all polysemous words.…”

“…BERT(Token-CLS) Since every target in a sentence needs to be disambiguated to find its exact sense, WSD task can be regarded as a tokenlevel classification task. To incorporate BERT to WSD task, we take the final hidden state of the token corresponding to the target word (if more than one token, we average them) and add a classification layer for every target lemma, which is the same as the last layer of the Bi-LSTM model (Kågebäck and Salomonsson, 2016).…”

Token Drop mechanism for Neural Machine Translation

“…Drop-Tag (Kågebäck and Salomonsson, 2016) replaces token with a < dropped > tag. The tag is subsequently treated just like any other word in the vocabulary and has a corresponding word embedding that is trained.…”

Word embeddings and recurrent neural networks based on Long-Short Term Memory nodes in supervised biomedical word sense disambiguation

“…Some advantages of using word embeddings is the lower dimensionality compared to bag-of-words and that words close in meaning are closer in the word em-bedding space. Very recent work still under preprint on using a special kind of recurrent network named LSTM (Long Short Term Memory) for WSD is recently being made available (Yuan et al, 2016) and with bidirectional LSTM (Kågebäck and Salomonsson, 2016), improving over more traditional supervised learning methods.…”