Mapping Unseen Words to Task-Trained Embedding Spaces

Abstract

We consider the supervised training setting in which we learn task-specific word embeddings. We assume that we start with initial embeddings learned from unlabelled data and update them to learn taskspecific embeddings for words in the supervised training data. However, for new words in the test set, we must use either their initial embeddings or a single unknown embedding, which often leads to errors. We address this by learning a neural network to map from initial embeddings to the task-specific embedding space, via a multi-loss objective function. The technique is general, but here we demonstrate its use for improved dependency parsing (especially for sentences with out-of-vocabulary words), as well as for downstream improvements on sentiment analysis.

Author Profiling for Hate Speech Detection

“…In all our methods, words not available in the GLoVe set are randomly initialized in the range ±0.05, indicating the lack of semantic information. By not mapping these words to a single random embedding, we mitigate against the errors that may arise due to their conflation (Madhyastha et al, 2015). A special OOV (out of vocabulary) token is also initialized in the same range.…”

Integrating Approaches to Word Representation

“…A tangential but noteworthy approach considers relations that are not curated in large graphs, but rather corpora annotated for inter-word relations such as syntactic dependencies(Madhyastha et al, 2016). Their system creates a mapping between a distributionally-obtained embedding table and one trained on the annotated parses, and generalizes this mapping to words which are now out-of-vocabulary for a further downstream task (e.g., sentiment analysis).…”