Discovering joint audio–visual codewords for video event detection

Jhuo, I-Hong; Ye, Guangnan; Gao, Shenghua; Liu, Dong; Jiang, Yu‐Gang; Lee, D. T.; Chang, Shih-Fu

doi:10.1007/s00138-013-0567-0

Cited by 45 publications

(42 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Xu et al [48] and Ye et al [50] adopted late fusion with specially designed methods to remove the noise of individually trained classifiers, and Jhuo et al used a joint audio-visual codebook for classification [16]. Our approach is fundamentally different from these state-of-the-art methods in its design and produces significantly higher performance.…”

Section: Results Of the Entire Frameworkmentioning

confidence: 94%

“…The approach was further enhanced in [18], where the temporal interaction of audio-visual features was investigated. Jhuo et al [16] improved the speed of training the audio-visual joint codebook by using standard local visual features like the SIFT, instead of the segmentation-based region features.…”

Section: Fusing Multiple Featuresmentioning

confidence: 99%

“…The average training time of each epoch for NN-EF, NN-LF and M-DBM are presented in Table 4, using a GPU-based implementation as Hollywood2 mAP CCV mAP Mathe et al [28] 61.0% Jiang et al [20] 59.5% Jain et al [15] 62.5% Jhuo et al [16] 64.0% Oneata et al [34] 63. 4% Xu et al [48] 60.3% Wang et al [46] 64.3% Ye et al [50] 64.0% Regularized DNN 65.7% Regularized DNN 70.6% Table 3: Comparison with state-of-the-art results in the literature.…”

Section: Computational Efficiencymentioning

confidence: 99%

See 2 more Smart Citations

Exploring Inter-feature and Inter-class Relationships with Deep Neural Networks for Video Classification

Jiang

Wang

et al. 2014

Proceedings of the 22nd ACM International Conference on Multimedia

Self Cite

128

View full text Add to dashboard Cite

Videos contain very rich semantics and are intrinsically multimodal. In this paper, we study the challenging task of classifying videos according to their high-level semantics such as human actions or complex events. Although extensive efforts have been paid to study this problem, most existing works combined multiple features using simple fusion strategies and neglected the exploration of inter-class semantic relationships. In this paper, we propose a novel unified framework that jointly learns feature relationships and exploits the class relationships for improved video classification performance. Specifically, these two types of relationships are learned and utilized by rigorously imposing regularizations in a deep neural network (DNN). Such a regularized DNN can be efficiently launched using a GPU implementation with an affordable training cost. Through arming the DNN with better capability of exploring both the interfeature and the inter-class relationships, the proposed regularized DNN is more suitable for identifying video semantics. With extensive experimental evaluations, we demonstrate that the proposed framework exhibits superior performance over several state-of-the-art approaches. On the well-known Hollywood2 and Columbia Consumer Video benchmarks, we obtain to-date the best reported results: 65.7% and 70.6% respectively in terms of mean average precision.

show abstract

Section: Results Of the Entire Frameworkmentioning

confidence: 94%

Section: Fusing Multiple Featuresmentioning

confidence: 99%

Section: Computational Efficiencymentioning

confidence: 99%

See 1 more Smart Citation

Exploring Inter-feature and Inter-class Relationships with Deep Neural Networks for Video Classification

Jiang

Wang

et al. 2014

Proceedings of the 22nd ACM International Conference on Multimedia

Self Cite

128

View full text Add to dashboard Cite

show abstract

“…Jiang et al [10] grouped visual and audio features together with their temporal relationship and computed combined features from these groups. Similarly, Jhou et al [13] constructed a bigraph with temporal concurrency between visual words and employed a k-way segmentation algorithm to combine visual and audio features. In this work, we propose to construct a spatio-temporal bigraph and use the k-way segmentation algorithm to combine multiple features.…”

Section: Related Workmentioning

confidence: 99%

“…Fernando et al [12] captured video-wide temporal information for action recognition. Jhou et al [13] proposed to use the concurrent statistical information to construct a bipartite graph for feature fusion. In fact, these methods use the temporal relationship between audio feature and visual feature for early fusion.…”

Section: Introductionmentioning

confidence: 99%

Spatio-temporal information for human action recognition

Yao

Liu

Huang

2016

J Image Video Proc.

View full text Add to dashboard Cite

Human activity recognition in videos is important for content-based videos indexing, intelligent monitoring, human-machine interaction, and virtual reality. This paper uses the low-level feature-based framework for human activity recognition which includes feature extraction and descriptor computing, early multi-feature fusion, video representation, and classification. This paper improves the first two steps. We propose a spatio-temporal bigraph-based multi-feature fusion algorithm to capture the useful visual information for recognition. Meanwhile, we introduce a compressed spatio-temporal video representation to bag of words representation. Our experiments on two popular datasets show efficient performance.

show abstract

Learning to Separate Object Sounds by Watching Unlabeled Video

Gao

Feris²,

Grauman

2018

Lecture Notes in Computer Science

254

222

View full text Add to dashboard Cite

Perceiving a scene most fully requires all the senses. Yet modeling how objects look and sound is challenging: most natural scenes and events contain multiple objects, and the audio track mixes all the sound sources together. We propose to learn audio-visual object models from unlabeled video, then exploit the visual context to perform audio source separation in novel videos. Our approach relies on a deep multi-instance multi-label learning framework to disentangle the audio frequency bases that map to individual visual objects, even without observing/hearing those objects in isolation. We show how the recovered disentangled bases can be used to guide audio source separation to obtain better-separated, object-level sounds. Our work is the first to learn audio source separation from large-scale "in the wild" videos containing multiple audio sources per video. We obtain state-of-the-art results on visuallyaided audio source separation and audio denoising. Our video results:

show abstract

Discovering joint audio–visual codewords for video event detection

Cited by 45 publications

References 18 publications

Exploring Inter-feature and Inter-class Relationships with Deep Neural Networks for Video Classification

Exploring Inter-feature and Inter-class Relationships with Deep Neural Networks for Video Classification

Spatio-temporal information for human action recognition

Learning to Separate Object Sounds by Watching Unlabeled Video

Contact Info

Product

Resources

About