Large-scale multimodal semantic concept detection for consumer video

Chang, Shih‐Fu; Ellis, Dan; Jiang, Wei; Lee, Keansub; Yanagawa, Akira; Loui, Alexander C.; Luo, Jiebo

doi:10.1145/1290082.1290118

Cited by 105 publications

(76 citation statements)

References 10 publications

(15 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…A few studies also included audio features [25], [26], [91] and found significant advantage in doing so. Multimodal multisource event detection is likely to receive more attention in the near future due to many emerging applications and the availability of large multimodal collections.…”

Section: ) Discussionmentioning

confidence: 99%

Event Mining in Multimedia Streams

2008

View full text Add to dashboard Cite

| Events are real-world occurrences that unfold over space and time. Event mining from multimedia streams improves the access and reuse of large media collections, and it has been an active area of research with notable recent progress. This paper contains a survey on the problems and solutions in event mining, approached from three aspects: event description, event-modeling components, and current event mining systems. We present a general characterization of multimedia events, motivated by the maxim of five BW[s and one BH[ for reporting real-world events in journalism: when, where, who, what, why, and how. We discuss the causes for semantic variability in real-world descriptions, including multilevel event semantics, implicit semantics facets, and the influence of context. We discuss five main aspects of an event detection system. These aspects are: the variants of tasks and event definitions that constrain system design, the media capture setup that collectively define the available data and necessary domain assumptions, the feature extraction step that converts the captured data into perceptually significant numeric or symbolic forms, statistical models that map the feature representations to richer semantic descriptions, and applications that use event metadata to help in different information-seeking tasks. We review current event-mining systems in detail, grouping them by the problem formulations and approaches. The review includes detection of events and actions in one or more continuous sequences, events in edited video streams, unsupervised event discovery, events in a collection of media objects, and a discussion on ongoing benchmark activities. These problems span a wide range of multimedia domains such as surveillance, meetings, broadcast news, sports, documentary, and films, as well as personal and online media collections. We conclude this survey with a brief outlook on open research directions.

show abstract

Section: ) Discussionmentioning

confidence: 99%

Event Mining in Multimedia Streams

2008

View full text Add to dashboard Cite

show abstract

“…We also use the challenging Kodak's consumer video data set provided in [13], [21] for evaluation. Unlike the Caltech images, content in this raw video source involves more variations in imaging conditions (view, scale, lighting) and scene complexity (background and number of objects).…”

Section: Performance Over Consumer Videosmentioning

confidence: 99%

“…To explore complementary features from both audio and visual channels, we extract similar features as [21]: visual features, e.g., grid color moments, Gabor texture, edge direction histogram, from keyframes, resulting in 346-dimension visual feature vectors; Mel-Frequency Cepstral Coefficients (MFCCs) from each audio frame (10ms) and delta MFCCs from neighboring frames. Over the video interval associated with each keyframe, the mean and covariance of the audio frame features are computed to generate a 2550-dimension audio feature vector .…”

Section: Performance Over Consumer Videosmentioning

confidence: 99%

Semantic Concept Classification by Joint Semi-supervised Learning of Feature Subspaces and Support Vector Machines

Jiang

Chang

Jebara

et al. 2008

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Abstract. The scarcity of labeled training data relative to the highdimensionality multi-modal features is one of the major obstacles for semantic concept classification of images and videos. Semi-supervised learning leverages the large amount of unlabeled data in developing effective classifiers. Feature subspace learning finds optimal feature subspaces for representing data and helping classification. In this paper, we present a novel algorithm, Locality Preserving Semi-supervised Support Vector Machines (LPSSVM), to jointly learn an optimal feature subspace as well as a large margin SVM classifier. Over both labeled and unlabeled data, an optimal feature subspace is learned that can maintain the smoothness of local neighborhoods as well as being discriminative for classification. Simultaneously, an SVM classifier is optimized in the learned feature subspace to have large margin. The resulting classifier can be readily used to handle unseen test data. Additionally, we show that the LPSSVM algorithm can be used in a Reproducing Kernel Hilbert Space for nonlinear classification. We extensively evaluate the proposed algorithm over four types of data sets: a toy problem, two UCI data sets, the Caltech 101 data set for image classification, and the challenging Kodak's consumer video data set for semantic concept detection. Promising results are obtained which clearly confirm the effectiveness of the proposed method.

show abstract

“…Previous work has explored detecting semantic concepts in multiple modalities to capture the embedded semantics in the multimedia stream [1] [2] [3]. In this paper, we limit ourselves to discuss the audio semantic concepts.…”

Section: Introductionmentioning

confidence: 99%

Semi-automatic audio semantic concept discovery for multimedia retrieval

Wang

Rawat

Metze

2014

2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Huge amount of videos on the Internet have rare textual information, which makes video retrieval challenging given a text query. Previous work explored semantic concepts for content analysis to assist retrieval. However, the humandefined concepts might fail to cover the data and there is a potential gap between these concepts and the semantics expected from user's query. Also, building a corpus is expensive and time-consuming. To address these issues, we propose a semi-automatic framework to discover the semantic concepts. We limit ourselves in audio modality here. In the paper, we also discuss how to select meaningful vocabulary from the discovered hierarchical sub-categories and provide an approach to detect all the concepts without further annotation. We evaluate the method on NIST 2011 multimedia event detection (MED) dataset.

show abstract

Large-scale multimodal semantic concept detection for consumer video

Cited by 105 publications

References 10 publications

Event Mining in Multimedia Streams

Event Mining in Multimedia Streams

Semantic Concept Classification by Joint Semi-supervised Learning of Feature Subspaces and Support Vector Machines

Semi-automatic audio semantic concept discovery for multimedia retrieval

Contact Info

Product

Resources

About