Web Multimedia Object Classification Using Cross-Domain Correlation Knowledge

Abstract

Abstract-Given a collection of web images with the corresponding textual descriptions, in this paper, we propose a novel cross-domain learning method to classify these web multimedia objects by transferring the correlation knowledge among different information sources. Here, the knowledge is extracted from unlabeled objects through unsupervised learning and applied to perform supervised classification tasks. To mine more meaningful correlation knowledge, instead of using commonly used visual words in the traditional bag-of-visual-words (BoW) model, we discover higher level visual components (words and phrases) to incorporate the spatial and semantic information into our image representation model, i.e., bag-of-visual-phrases (BoP). By combining the enriched visual components with the textual words, we calculate the frequently co-occurring pairs among them to construct a cross-domain correlated graph in which the correlation knowledge is mined. After that, we investigate two different strategies to apply such knowledge to enrich the feature space where the supervised classification is performed. By transferring such knowledge, our cross-domain transfer learning method can not only handle large scale web multimedia objects, but also deal with the situation that the textual descriptions of a small portion of web images are missing. Empirical experiments on two different datasets of web multimedia objects are conducted to demonstrate the efficacy and effectiveness of our proposed cross-domain transfer learning method.

Cross-Domain Feature Learning in Multimedia

“…A knowledge adaptation method for Ad Hoc multimedia event detection is proposed in [32]. In [33], cross-domain correlation knowledge is used for web multimedia object classification. In [34], a feature transformation method is proposed to indirectly transfer semantic knowledge between text and images.…”

Image-Text Joint Learning for Social Images with Spatial Relation Model

“…Significant efforts have been taken to integrate visual and textual analyses [2][3][4]. For example, Wang et al [5] present an algorithm to learn the relations between scenes, objects, and texts with the help of image-level labels.…”

“…(1) Visual content and text are always separately learned, making the traditional methods hard to be trained end-to-end. (2) Learning tasks converted to classification problems, empowered by large-scale annotated data with end-to-end training using neural networks, which is not capable of describing concepts unseen in the training pairs. (3) e spatial relations defined by prepositions have to be learned with the bounding boxes of objects, which are so immoderately challenging to obtain.…”

Harvesting Visual Objects from Internet Images via Deep-Learning-Based Objectness Assessment

“…The rapid growth of internet images also brings up many applications that make a good use of this big data, such as image retrieval [33], stitching [29], recommendation [62], image quality assessment [17], or benchmark dataset making [58], etc. It is also obvious that within these images the number and richness of contained objects are of great usefulness, especially in some data-driven applications related to object manipulation, such as object retrieval [51,57], classification [32], enhancement [59], etc. Traditionally, searching for images that contain desired objects is done through online search engines such as Google Images, which utilizes content-based image retrieval techniques to find images that contain similar objects, or simply acquires images by their tagged information.…”