Combining Language and Vision with a Multimodal Skip-gram Model

Abstract

We extend the SKIP-GRAM model of Mikolov et al. (2013a) by taking visual information into account. Like SKIP-GRAM, our multimodal models (MMSKIP-GRAM) build vector-based word representations by learning to predict linguistic contexts in text corpora. However, for a restricted set of words, the models are also exposed to visual representations of the objects they denote (extracted from natural images), and must predict linguistic and visual features jointly. The MMSKIP-GRAM models achieve good performance on a variety of semantic benchmarks. Moreover, since they propagate visual information to all words, we use them to improve image labeling and retrieval in the zero-shot setup, where the test concepts are never seen during model training. Finally, the MMSKIP-GRAM models discover intriguing visual properties of abstract words, paving the way to realistic implementations of embodied theories of meaning.

Visually Grounded Meaning Representations

“…On the other hand, the textual attributes fall short compared to the skip-gram embeddings (tAttrib vs. skip-gram, T). The bimodal SAE trained on the latter (skip-gram, vAttrib; T+V) is the overall best model, outperforming SVD and CCA (skip-gram, vAttrib, T+V), Lazaridou et al [40], Bruni et al [37], and all concatenation models. It yields a correlation coefficient of ρ =.77 on semantic similarity and ρ = 0.66 on visual similarity.…”

“…We report results with our SAE model, SVD, and the CCA models using (automatically obtained) textual and visual attributes (tAttrib, vAttrib) or skip-gram embeddings and visual attributes (skip-gram, vAttrib). We also compare SAE to Lazaridou et al's [40] multimodal skip-gram model and Bruni et al [37]. The third section of the table presents concatenation models using our textual and visual attributes (tAttrib, vAttrib), skip-gram embeddings, CNN features, and combinations thereof.…”

“…Several models ( [15], [19], [22]) present extensions of Latent Dirich-let Allocation (LDA, [42]) where topic distributions are learned from words and other perceptual units treating them both as observed variables. Hill and Korhonen [34] extend the skip-gram network model [33] in a similar fashion, perceptual input is encoded verbally and treated as a word's linguistic context, whereas Lazaridou et al [40] modify skipgram's learning objective so that representations are trained to predict linguistic and visual features. In most cases the visual and textual modalities are decoupled and obtained independently, i.e., from text corpora and feature norms or image databases (but see [19] for an exception).…”

“…Lazaridou et al's [40] model takes into account visual information during training by adding a visual objective 17 to the text-based skip-gram objective. The visual objective is to maximize the similarity between the fixed visual vector of a target word (when available) and its textual embedding to be learned using a max-margin framework.…”

“…Drawing inspiration from the successful application of attribute classifiers in object recognition, Silberer et al [21] show that automatically predicted visual attributes from images can act as substitutes for feature norms without any critical information loss. In other work ( [39], [40]) representations for the visual modality are obtained directly from image pixels using the feature extraction layers of a deep convolutional neural network (CNN) trained on a large labeled object recognition data set. Finally, some models use human generated image tags as a proxy for visual information ( [20], [34]).…”