Cross-modal Retrieval and Synthesis (X-MRS): Closing the Modality Gap in Shared Subspace Learning

Abstract

Computational food analysis (CFA) naturally requires multi-modal evidence of a particular food, e.g., images, recipe text, etc. A key to making CFA possible is multi-modal shared representation learning, which aims to create a joint representation of the multiple views (text and image) of the data. In this work we propose a method for food domain cross-modal shared representation learning that preserves the vast semantic richness present in the food data. Our proposed method employs an effective transformer-based multilingual recipe encoder coupled with a traditional image embedding architecture. Here, we propose the use of imperfect multilingual translations to effectively regularize the model while at the same time adding functionality across multiple languages and alphabets. Experimental analysis on the public Recipe1M dataset shows that the representation learned via the proposed method significantly outperforms the current state-of-the-arts (SOTA) on retrieval tasks. Furthermore, the representational power of the learned representation is demonstrated through a generative food image synthesis model conditioned on recipe embeddings. Synthesized images can effectively reproduce the visual appearance of paired samples, indicating that the learned representation captures the joint semantics of both the textual recipe and its visual content, thus narrowing the modality gap. CCS CONCEPTS• Information systems → Multimedia and multimodal retrieval; • Computing methodologies → Learning latent representations.

Transformer-based Cross-Modal Recipe Embeddings with Large Batch Training

“…However, Ours+ResNet obtained much higher R@{5,10} than RDE-GAN. Some metrics on 1k test set size (R@{5,10} in Image-to-Recipe and R@10 in Recipe-to-Image) are lower than X-MRS [9] and H-T [26]. Since the difference is less than 1.0 and Ours+ResNet has much higher R@1 (more than 2.0) and better medR (2.4 in medR), we suppose the proposed method with ResNet-50 image encoder backbone still outperforms the previous works.…”

“…MCEN [6] introduced cross-modal attention and consistency and Zan et al [33] introduced BERT [4] as a recipe encoder to enable cross-modal retrieval. X-MRS [9] introduced a Transformer [30] encoder to gain recipe embedding, further proposed the use of imperfect multilingual translations, and achieved state-of-the-art performances on retrieval tasks. Salvador et al proposed a simply but effective framework H-T [26], to facilitate the power of Transformer.…”

“…GAN [7] has been introduced and proven its effectiveness for improving recipe retrieval performance in some recent works [9,29,31,36]. R2GAN [36], ACME [31] and X-MRS [9] applied textconditioned image synthesis which generates recipe images from recipe embeddings, while RDE-GAN [29] proposed to disentangle image features into dish shape features, which contained only nonrecipe information, and recipe embeddings and to integrate both of them to generate recipe images.…”

“…We also evaluated the performance of image generation conditioned on recipe embeddings using Fréchet Inception Distance (FID) [15] score following previous works [9,29] measuring the similarity of the distribution between generated images and groundtruth images. Training Details.…”

“…MCEN [6] applied attention mechanism, and Zan et al [33] applied BERT [4] for recipe encoding. With the trend of Transformer [30] in natural language processing, X-MRS [9] and H-T [26] were recently proposed to adopt this technique in recipe retrieval. The research H-T in particular introduced a simple but effective framework with a Transformer-based structure recipe encoder and self-supervised learning, allowing the model to explore complementary information among recipe texts.…”