MAiVAR: Multimodal Audio-Image and Video Action Recognizer

Abstract

In line with the human capacity to perceive the world by simultaneously processing and integrating highdimensional inputs from multiple modalities like vision and audio, we propose a novel model, MAiVAR-T (Multimodal Audio-Image to Video Action Recognition Transformer). This model employs an intuitive approach for the combination of audio-image and video modalities, with a primary aim to escalate the effectiveness of multimodal human action recognition (MHAR). At the core of MAiVAR-T lies the significance of distilling substantial representations from the audio modality and transmuting these into the image domain. Subsequently, this audio-image depiction is fused with the video modality to formulate a unified representation. This concerted approach strives to exploit the contextual richness inherent in both audio and video modalities, thereby promoting action recognition. In contrast to existing state-of-the-art strategies that focus solely on audio or video modalities, MAiVAR-T demonstrates superior performance. Our extensive empirical evaluations conducted on a benchmark action recognition dataset corroborate the model's remarkable performance. This underscores the potential enhancements derived from integrating audio and video modalities for action recognition purposes.

MHAiR: A Dataset of Audio-Image Representations for Multimodal Human Actions

“…This dataset can thus be used to analyze critical features from the action sequences in the image form. This dataset extends our previous publication [2] which outperforms state-of-the-art methods producing an accuracy of 91.2% by focusing on multimodal representations of action sequences to present critical features in audio from different perspectives, as captured from each action sample. These datasets were also used as a pre-requisite requirement in developing an intelligent multimodal action recognition system for classifying actions using deep learning algorithms based on acoustic and video modality.…”

“…In the context of multimodal action recognition, as in Multimodal Audio-image and Video Action Recognition (MAiVAR) framework [2], these data are utilized, and they demonstrate superior performance compared to other audio representations. The study establishes a benchmark approach for using this dataset.…”

“…For example, a higher spectral centroid value often corresponds to a "brighter" or "sharper" sound, while a lower spectral centroid value usually indicates a "duller" or "muddier" sound [17]. By converting the spectral centroid over time into an image, we can capture spatial and temporal information that can be effectively processed by deep learning models [2]. Spectral centroid can also help in distinguishing actions based on their tonal or harmonic characteristics.…”

Multiscale knowledge distillation with attention based fusion for robust human activity recognition

“…Bruce et al 48 devised a model-level multimodal fusion approach, employing spatiotemporal graph convolutional networks as the skeletal modality to learn how to transfer attention weights from the skeletal modality to the RGB modality network. Shaikh et al 49 proposed a meaningful representation extraction method for image and audio and fused it with video representation, demonstrating better performance in activity recognition compared to single-modality audio and video. The first two methods primarily rely on multihead self-attention mechanism-based multimodal recognition methods, enabling the global learning of relationships between different modalities.…”

Multimodal Fusion for Audio-Image and Video Action Recognition

“…The outputs of the last convolution layer of these two separate network streams are processed to the fully connected layer. This step predicts the final output after fusing the classification scores and considers individual class labels at the score layer [105]. Different decision rules are deployed to fuse the scores at this stage.…”