Data Modeling Strategies for Imbalanced Learning in Visual Search

This survey aims at providing multimedia researchers with a state-of-the-art overview of fusion strategies, which are used for combining multiple modalities in order to accomplish various multimedia analysis tasks. The existing literature on multimodal fusion research is presented through several classifications based on the fusion methodology and the level of fusion (feature, decision, and hybrid). The fusion methods are described from the perspective of the basic concept, advantages, weaknesses, and their usage in various analysis tasks as reported in the literature. Moreover, several distinctive issues that influence a multimodal fusion process such as, the use of correlation and independence, confidence level, contextual information, synchronization between different modalities, and the optimal modality selection are also highlighted. Finally, we present the open issues for further research in the area of multimodal fusion.

show abstract

“…For instance, dealing with unlabeled data in fusion [130] and handling noisy positive data for fusion [54].…”

Section: Other Considerationsmentioning

confidence: 99%

Multimodal fusion for multimedia analysis: a survey

et al. 2010

View full text Add to dashboard Cite

show abstract

“…On the other hand, current supervised reranking approaches [6] [7] usually treat the query examples as "positive" and sample the low-ranked samples in the initial search results as "pseudo-negative." Then, a set of visual features are extracted to build a new search model and produce the visualbased search results.…”

Section: Related Workmentioning

confidence: 99%

“…Conventional reranking approaches to video search have proceeded along two dimensions: (1) mining the meaningful information from the initial ranked list to perform unsupervised reranking [2] [3] [4] [5], and (2) leveraging the auxiliary knowledge to reorder the samples in a supervised way [6] [7]. Typical unsupervised approaches focus on mining the relevant or irrelevant information in the initial search results which is usually obtained from the text-based search systems.…”

Section: Introductionmentioning

confidence: 99%

Optimizing video search reranking via minimum incremental information loss

Liu

Mei

et al. 2008

Proceedings of the 1st ACM International Conference on Multimedia Information Retrieval

View full text Add to dashboard Cite

This paper is concerned with video search reranking -the task of reordering the initial ranked documents (video shots) to improve the search performance -in an optimization framework. Conventional supervised reranking approaches empirically convert the reranking as a classification problem in which each document is determined relevant or not, followed by reordering the documents according to the confidence scores of classification. We argue that reranking is essentially an optimization problem in which the ranked list is globally optimal if any two arbitrary documents from the list are correctly ranked in terms of relevance, rather than simply classifying a document into relevant or not. Therefore, we propose in this paper to directly optimize video search reranking from a novel viewpoint of information theory, that is, to identify an optimal set of correctly-ranked document pairs which maximally preserves the relevant information and simultaneously carries the irrelevant information as little as possible. The final reranked list is then directly recovered from this optimal set of pairs. Under the framework, we further propose an effective algorithm, called minimum incremental information loss (MIIL) reranking, to solve the optimization problem more practically. We conducted comprehensive experiments on automatic video search task over TRECVID 2005-2007 benchmarks, and showed significant and consistent improvements over the text search baseline and other reranking approaches.

show abstract

“…Multi-bag SVM [11] is used to produce the initial results based on these features. Then we refine the baselines through the proposed MRTD-based re-ranking scheme.…”

Section: Motion Fusion and Re-rankingmentioning

confidence: 99%

Motion region-based trajectory analysis and re-ranking for video retrieval

Feng

Cao

Lin

et al. 2009

2009 IEEE International Conference on Multimedia and Expo

View full text Add to dashboard Cite

Event-related query is playing a more and more important role in video retrieval. However, it is still a challenge to the existing video retrieval engines for lacking the effective motion analysis. In this paper, we propose a novel re-ranking scheme for video retrieval based on motion region trajectory analysis. By focusing on the changes of the primary moving regions, we construct an intuitive motion region-based trajectory descriptor (MRTD) to depict the shot activities. In the re-ranking phase, the proposed approach takes the MRTD as a motion cue and re-ranks the baseline results by motion-related query selection and MRTD-based weighting. We evaluate our method in TRECVID2007 and 2008 datasets, and observe consistent improvement over all the baselines, leading to a greatest performance gain of 42.9%, and an average gain of 17%. The experiments also show that the motion descriptor of MRTD is fruitful for a variety of features.

show abstract

Data Modeling Strategies for Imbalanced Learning in Visual Search

Cited by 12 publications

References 7 publications

Multimodal fusion for multimedia analysis: a survey

Multimodal fusion for multimedia analysis: a survey

Optimizing video search reranking via minimum incremental information loss

Motion region-based trajectory analysis and re-ranking for video retrieval

Contact Info

Product

Resources

About