Multiple feature fusion for social media applications

Abstract

The emergence of social media as a crucial paradigm has posed new challenges to the research and industry communities, where media are designed to be disseminated through social interaction. Recent literature has noted the generality of multiple features in the social media environment, such as textual, visual and user information. However, most of the studies employ only a relatively simple mechanism to merge the features rather than fully exploit feature correlation for social media applications. In this paper, we propose a novel approach to fusing multiple features and their correlations for similarity evaluation. Specifically, we first build a Feature Interaction Graph (FIG) by taking features as nodes and the correlations between them as edges. Then, we employ a probabilistic model based on Markov Random Field to describe the graph for similarity measure between multimedia objects. Using that, we design an efficient retrieval algorithm for large social media data. Further, we integrate temporal information into the probabilistic model for social media recommendation. We evaluate our approach using a large real-life corpus collected from Flickr, and the experimental results indicate the superiority of our proposed method over stateof-the-art techniques.

Integrating Deep Learning with Correlation-based Multimedia Semantic Concept Detection

“…Usually, either features or modalities will be fused based on their ability to retrieve semantic concepts [17,38,39,43,145]. is built from visual and audio modalities, which is later partitioned into bi-modal words that can be also considered as joint patterns across modalities.…”

Multi-level Video Filtering Using Non-textual Contents

“…However, the filtering models we introduced are naturally oriented to single content type. Accordingly, many studies on multiple content fusion has been conducted in recent years [17,76,99]. Based on existing works, there are two branches of fusion strategies, one is early fusion which applies fusion on the inputs of the filtering models, and the other one is late fusion which applies fusion on the outputs of the filtering models.…”

“…It has been widely used in document retrieval. In [17], a feature correlation tree is constructed to model the relations between different content vectors. Then, conditional random field method was applied on the tree to perform the late fusion.…”

“…Recent studies [17,4,60] show that both fusion strategies push the video-level filtering accuracy in to a high level. Sometimes, the high accuracy means the highly ranked ones could be used as exemplar videos.…”

“…In recent years, designing fusion strategies of multiple features has become a major research trend and different techniques have been proposed. There are two widely accepted yet independent strategies to fuse multiple features [17]: early fusion and late fusion. Most works on early fusion try to map multiple feature spaces to a unified one.…”