Learning the semantics of multimedia queries and concepts from a small number of examples

Abstract

In this paper we unify two supposedly distinct tasks in multimedia retrieval. One task involves answering queries with a few examples. The other involves learning models for semantic concepts, also with a few examples. In our view these two tasks are identical with the only differentiation being the number of examples that are available for training. Once we adopt this unified view, we then apply identical techniques for solving both problems and evaluate the performance using the NIST TRECVID benchmark evaluation data [15]. We propose a combination hypothesis of two complementary classes of techniques, a nearest neighbor model using only positive examples and a discriminative support vector machine model using both positive and negative examples. In case of queries, where negative examples are rarely provided to seed the search, we create pseudo-negative samples. We then combine the ranked lists generated by evaluating the test database using both methods, to create a final ranked list of retrieved multimedia items. We evaluate this approach for rare concept and query topic modeling using the NIST TRECVID video corpus.In both tasks we find that applying the combination hypothesis across both modeling techniques and a variety of features results in enhanced performance over any of the baseline models, as well as in improved robustness with respect to training examples and visual features. In particular, we observe an improvement of 6% for rare concept detection and 17% for the search task.

Semantic Labeling of Multimedia Content Clusters

“…The dataset consists of 308 video clips of vacation videos, with 19,238 extracted shots and representative keyframes. Clusters:We extract most descriptive visual features [6] from the keyframes. We use K-means algorithm to cluster the whole dataset into 100 visually distinctive clusters in the localized color space.…”

“…We use K-means algorithm to cluster the whole dataset into 100 visually distinctive clusters in the localized color space. Localized color descriptor is extracted from a 5x5 image grid and is represented by the first 3 moments for each grid tile in the LAB color space [6].…”

“…Such inductive inference of concept from a cluster, describes the content of that cluster well. We extract most descriptive visual features [6] from the representative keyframes. We associate score with each data item for each semantic concept.…”

Hybrid negative example selection using visual and conceptual features

“…In the field of image/video annotation and retrieval, many researchers have simply used randomly selected unlabeled examples as negative examples [12,13,18,20]. However, there is no guarantee that the use of such negative examples will lead to accurate retrieval.…”

Integrating Deep Learning with Correlation-based Multimedia Semantic Concept Detection

“…Usually, either features or modalities will be fused based on their ability to retrieve semantic concepts [17,38,39,43,145]. is built from visual and audio modalities, which is later partitioned into bi-modal words that can be also considered as joint patterns across modalities.…”