Multimodal Word Discovery and Retrieval with Phone Sequence and Image Concepts

Abstract

This paper demonstrates three different systems capable of performing the multimodal word discovery task. A multimodal word discovery system accepts, as input, a database of spoken descriptions of images (or a set of corresponding phone transcripts), and learns a lexicon which is a mapping from phone strings to their associated image concepts. Three systems are demonstrated: one based on a statistical machine translation (SMT) model, two based on neural machine translation (NMT). On Flickr8k, the SMT-based model performs much better than the NMT-based one, achieving a 49.6% F1 score. Finally, we apply our word discovery system to the task of image retrieval and achieve 29.1% recall@10 on the standard 1000image Flickr8k tests set.

A Study on the Construction of the National Media Image of American Science Fiction Films in the New Century

“…In other words, the national media image acts as a bridge between the national physical image and the cognitive image. [5] From an information dissemination perspective, the national media image essentially involves the construction and dissemination of information related to the national image. In this context, "information" specifically refers to the concrete content conveyed by the abstract symbol of the national image.…”