Learning of Multimodal Representations With Random Walks on the Click Graph

Abstract

In multimedia information retrieval, most classic approaches tend to represent different modalities of media in the same feature space. With the click data collected from the users' searching behavior, existing approaches take either one-to-one paired data (text-image pairs) or ranking examples (text-query-image and/or image-query-text ranking lists) as training examples, which do not make full use of the click data, particularly the implicit connections among the data objects. In this paper, we treat the click data as a large click graph, in which vertices are images/text queries and edges indicate the clicks between an image and a query. We consider learning a multimodal representation from the perspective of encoding the explicit/implicit relevance relationship between the vertices in the click graph. By minimizing both the truncated random walk loss as well as the distance between the learned representation of vertices and their corresponding deep neural network output, the proposed model which is named multimodal random walk neural network (MRW-NN) can be applied to not only learn robust representation of the existing multimodal data in the click graph, but also deal with the unseen queries and images to support cross-modal retrieval. We evaluate the latent representation learned by MRW-NN on a public large-scale click log data set Clickture and further show that MRW-NN achieves much better cross-modal retrieval performance on the unseen queries/images than the other state-of-the-art methods.

A Comprehensive Survey of Graph Embedding: Problems, Techniques, and Applications

“…It exploits user-image links to embed users and images into the same space so that they can be directly compared for image recommendation. In [36], a click graph is considered which contains images and text queries. The image-query edge indicates a click of an image given a query, where the click count serves as the edge weight.…”

“…DGK [93] graphlet kernel: random sampling [114] 2 nd (by graphlet) SkipGram (Eqs. 11-12 ) metapath2vec [46] meta-path based random walk 2 nd heterogeneous SkipGram ProxEmbed [44] truncate random walk node ranking tuples HSNL [29] truncate random walk 2 nd + QA ranking tuples LSTM RMNL [30] truncated random walk 2 nd + user-question quality ranking DeepCas [63] Markov chain based random walk information cascade sequence GRU MRW-MN [36] truncated random walk 2 nd + cross-modal feature difference DCNN+ SkipGram sequence of nodes as a fixed length vector, e.g., represent a sentence (i.e., a sequence of words) as one vector. LSTM is then adopted in such scenarios to embed a node sequence.…”

“…Moreover, it can convert data instances from different sources/platforms/views into one common space so that they are directly comparable. For example, [16], [34], [36] use graph embedding for cross-modal retrieval, such as contentbased image retrieval, keyword-based image/video search. The advantages of using graph embedding for representation learning is that the graph manifold of the training data instances are preserved in the representations and can further benefit the follow-up applications.…”

Visir

“…Figure 1 (b) illustrates this point: there is only a single tag happiness and none for the concrete objects in the image. For the big search engines, one way of overcoming this bottleneck is to exploit query-and-click logs (e.g., [6,17,45]). The query keyword(s) associated with a click can be treated as label(s) for the clicked image.…”

Cross-Media Similarity Evaluation for Web Image Retrieval in the Wild

“…Two, represent queries in a visual space by learning visual templates of the queries [19], [20], and compute the cross-media similarity by visual matching. Third, also the main stream, build a common latent space, e.g., by maximizing correlation between relevant image-query pairs [10], [11], [13], [21] or by minimizing a ranking based loss [7], [9], [12], [22]. Given the progress described above, an important question then is Q1.…”