Hubness and Pollution: Delving into Cross-Space Mapping for Zero-Shot Learning

Abstract

Zero-shot methods in language, vision and other domains rely on a cross-space mapping function that projects vectors from the relevant feature space (e.g., visualfeature-based image representations) to a large semantic word space (induced in an unsupervised way from corpus data), where the entities of interest (e.g., objects images depict) are labeled with the words associated to the nearest neighbours of the mapped vectors. Zero-shot cross-space mapping methods hold great promise as a way to scale up annotation tasks well beyond the labels in the training data (e.g., recognizing objects that were never seen in training). However, the current performance of cross-space mapping functions is still quite low, so that the strategy is not yet usable in practical applications. In this paper, we explore some general properties, both theoretical and empirical, of the cross-space mapping function, and we build on them to propose better methods to estimate it. In this way, we attain large improvements over the state of the art, both in cross-linguistic (word translation) and cross-modal (image labeling) zero-shot experiments.

Knowledge Distillation for Bilingual Dictionary Induction

“…• The method Lazaridou et al was proposed by (Lazaridou et al, 2015). It uses a maxmargin ranking function and introduces a way of picking negative examples in computing the loss.…”

“…To address the first of our evaluation questions, we performed experiments on the dataset introduced by (Dinu et al, 2014), where the state-of-the art is the work of (Lazaridou et al, 2015). This is an Italian to English dataset, which consists of 5K translation pairs as training data, and 1.5K pairs as test data.…”

“…Our work is most related to models that do zero-shot learning for bilingual dictionary induction, using maps between vector spaces with seed dictionaries as training data. Examples include the models of (Mikolov et al, 2013;Dinu et al, 2014;Lazaridou et al, 2015;Vulic and Korhonen, 2016). Like these approaches, we first learn word embeddings for each language, then use a seed dictionary to train a mapping function between the two vector spaces.…”

“…Instead of the least squares loss shown in equation 2, we use a ranking loss, as in (Lazaridou et al, 2015), which aims to rank correct training data pairs (x i , y i ) higher than incorrect pairs (x i , y j ) with a margin of at least γ. The margin γ is a hyper-parameter which is application specific, and the incorrect labels, y j can be selected randomly such that j = i or in a more application specific manner 2 .…”

“…However, the state-of-the-art accuracy on zero-shot bilingual dictionary induction is still low. On the task of English to Italian (en → it), top-1 and top-10 accuracies are around 40% and 60%, respectively (Lazaridou et al, 2015;Dinu et al, 2014).…”