Learning Visually Grounded Sentence Representations

Abstract

We investigate grounded sentence representations, where we train a sentence encoder to predict the image features of a given captioni.e., we try to "imagine" how a sentence would be depicted visually-and use the resultant features as sentence representations. We examine the quality of the learned representations on a variety of standard sentence representation quality benchmarks, showing improved performance for grounded models over non-grounded ones. In addition, we thoroughly analyze the extent to which grounding contributes to improved performance, and show that the system also learns improved word embeddings.

Language Learning Using Speech to Image Retrieval

“…These textbased embeddings are trained to encode word-level semantic knowledge and have become a mainstay in work on sentence representations (e.g. [6,7]). When we want to learn language directly from speech, we will have to do so in a more end-to-end fashion, without prior lexical level knowledge in terms of both form and semantics.…”

“…In previous work [8] we used image-caption retrieval, where given a written caption the model must return the matching image and vice versa. We trained deep neural networks (DNNs) to create sentence embeddings without the use of prior knowledge of lexical semantics (see [7,9,10] for other studies on this task). The visually grounded sentence embeddings that arose capture semantic information about the sentence as measured by the Semantic Textual Similarity task (see [11]), performing comparably to text-only methods that require word embeddings.…”

Learning semantic sentence representations from visually grounded language without lexical knowledge

“…Recently, there have also been successful sentence encoder models which are trained on a supervised task and then transferred to other tasks (e.g. [10,11,12]).…”

“…So far, existing sentence embedding methods often require (pretrained) word embeddings [10,12], large amounts of data [8], or both [13,11]. While word embeddings are successful at enhancing sentence embeddings, they are not very plausible as a model of human language learning.…”

“…For sentence semantics, the multimodal task of image-caption retrieval, where given a caption the model must return the matching image and vice versa, has been proposed as a way of grounding sentence representations in vision [20,21]. Recently [12] found that such models do indeed produce embeddings that are useful in tasks like natural language inference, sentiment analysis and subjectivity/objectivity classification.…”