Exploring multimodal video representation for action recognition

Abstract

A video contains rich perceptual information, such as visual appearance, motion and audio, which can be used for understanding the activities in videos. Recent works have shown the combination of appearance (spatial) and motion (temporal) clues can significantly improve human action recognition performance in videos. To further explore the multimodal representation of video in action recognition, We propose a framework to learn a multimodal representations from video appearance, motion as well as audio data. Convolutional Neural Networks (CNN) are trained for each modality respectively. For fusing multiple features extracted with CNNs, we propose to add a fusion layer on the top of CNNs to learn a joint video representation. In fusion phase, we investigate both early fusion and late fusion with Neural Network and Support Vector Machine. Compare to existing works, (1) our work measures the benefits of taking audio information into consideration and (2) implements sophisticated fusion methods. The effectiveness of proposed approach is evaluated on UCF101 and UCF101-50 (selected subset in which each video contains audio data) for action recognition. The experimental results show that different modalities are complementary to each other and multimodal representation can be beneficial for final prediction. Furthermore, proposed fusion approach achieves 85.1% accuracy in fusing spatial-temporal on UCF101 (split 1), which is very competitive to state-of-the-art works.

A methodology for image annotation of human actions in videos

“…The authors of this study cater to the key limitation of Deconvolution layer that suffers from a checkerboard artifact problem the neural network is used as a source of a semi-supervised method for annotation. The Zheng et al [30], proposed MMDF-LDA: an improved multimodal latent Dirichlet allocation model for social image annotation. The authors focus on developing a data fusion model for social image annotation.…”

“…Wang et al [30], works on retrieving the perceptual information present inside videos. The information was based on human action recognition as spatial temporal constrains have made significant contribution on it.…”

Simulation-Driven Design of Smart Gloves for Gesture Recognition

“…Audio recording that accompanies the visual stream provides complementary information to appearance and motion information; for example, specific actions may be characterized by their unique sounds. Combining these two modalities within a deep learning pipeline at either the data level or the feature level has been thoroughly studied in many works [137,105,50,69], benefiting from the fact that video cameras typically provide both visual and audio streams simultaneously. Because video and audio data are largely heterogeneous, most of the fusion methods are represented by either features fusion or scores fusion.…”

“…Because video and audio data are largely heterogeneous, most of the fusion methods are represented by either features fusion or scores fusion. A three-pathways network was proposed to combine RGB frames, optical flow, and audio signal and demonstrated that simple feature fusion performed better than late scores fusion [137]. A two-stream CNN was trained in a self-supervised manner to capture the temporal alignment between the audio and video frames [105].…”

“…A Temporal Binding Network (TBN) [69] fuses multimodal input sequences within a binding window. As in [137] the authors combined audio, RGB frames, and optical flow from egocentric videos in a late feature fusion fashion and applied temporal aggregation to the extracted features for the final classification. The TBNs achieved state-of-the-art results on EPIC-Kitchens egocentric videos, demonstrating the importance of audio in egocentric vision.…”