Discriminative model fusion for semantic concept detection and annotation in video

Abstract

In this paper we describe a general information fusion algorithm that can be used to incorporate multimodal cues in building user-defined semantic concept models. We compare this technique with a Bayesian Network-based approach on a semantic concept detection task. Results indicate that this technique yields superior performance. We demonstrate this approach further by building classifiers of arbitrary concepts in a score space defined by a pre-deployed set of multimodal concepts. Results show annotation for user-defined concepts both in and outside the pre-deployed set is competitive with our best video-only models on the TREC Video 2002 corpus.

Exploiting Concept Association to Boost Multimedia Semantic Concept Detection

“…The subset is used in the evaluation of high-level feature extraction task in TRECVID 2006 2 . The names and the identity number (ID) of the semantic concepts are listed in Table 1.…”

“…Thus, N classifier are trained, each for one concept. Following the terms in [2], each classifier is treated as a basis model which plays a similar role as the eigenvector in the eigenspace, and it maps the low-level feature into one component in model score space. Traditionally, the N basis models are equally treated, and the feature dimension in the MBT fusion will be N. In Figure 1, the association of a target concept with other concepts varies in a large range from one concept to another.…”

“…Suppose we have M types of features and N concepts, M*N classifiers should be trained. These classifiers are treated as the bases to map a training sample into M*N-dimensional model score space [1,2,7]. Based on the model score space representation, another classifier is trained for each concept to reach the final classification decision.…”

“…In the classification stage, a test sample is first mapped into a M*Ndimensional model space vector, and then the final decision is made using the classifier trained on the model score space. This scheme has been proven successful by all systems developed for multimedia semantic concept detection and search in TRECVID 1 [1,2] (see TRECVID workshop papers for details).…”

Analysis of multimodal fusion techniques for audio-visual speech recognition

“…Существует множество приложений, в которых производится объединение аудио и видео, такие как распознавание речи [7][8][9][10][11][12][13], распознавание диктора [14][15][16], биометрическая верификация [17][18][19][20][21], обнаружение события [22][23][24][25], слежение за человеком или объектом [26][27][28][29][30][31], локализация и слежение за активным диктором [32,33], анализ музыкального контента, распознавание эмоций, видеопоиск, челове-ко-машинное взаимодействие, обнаружение голосовой активности и разделение источников звукового сигнала [34][35][36]. Очевидно, что в некоторых приложениях используются изображения лиц, а иногда даже движения всего тела, а не только лица.…”