Hear Me Out: Fusional Approaches for Audio Augmented Temporal Action Localization

Abstract

State of the art architectures for untrimmed video Temporal Action Localization (TAL) have only considered RGB and Flow modalities, leaving the information-rich audio modality totally unexploited. Audio fusion has been explored for the related but arguably easier problem of trimmed (clip-level) action recognition. However, TAL poses a unique set of challenges. In this paper, we propose simple but effective fusion-based approaches for TAL. To the best of our knowledge, our work is the first to jointly consider audio and video modalities for supervised TAL. We experimentally show that our schemes consistently improve performance for state of the art video-only TAL approaches. Specifically, they help achieve new state of the art performance on largescale benchmark datasets -

Do You Really Mean That? Content Driven Audio-Visual Deepfake Dataset and Multimodal Method for Temporal Forgery Localization

“…Bagchi et al [2] divided the methods for segment proposal estimation in temporal action localization into two main categories: methods based on anchors and methods based on predicting the boundary probabilities. As for the anchor-based, these methods mainly use sliding windows in the video, such as S-CNN [59], CDC [58], TURN-TAP [18] and CTAP [17].…”

“…It is worth noting that all these methods are unimodal, which is not optimal for the task of temporal forgery detection. The importance of multimodality was demonstrated recently by AVFusion [2]. Proposed Approach.…”

“…For the task of temporal forgery detection, both the audio and visual information are important, in addition to the required precise boundary proposals. In this paper, we introduce a multimodal method based on boundary probabilities and compare the performance with BMN [39], AGT [46], MDS [11] and AVFusion [2].…”

OWL (Observe, Watch, Listen): Localizing Actions in Egocentric Video via Audiovisual Temporal Context

“…[39] proposes a new task of audiovisual event localization that aims at predicting the event class from a 10second clip. [4] studies multi-modal fusion approaches for audiovisual localization but ablates it on THUMOS14 and ActivityNet . Compared to them, we design our method for long, diverse egocentric videos.…”

SVIP: Sequence VerIfication for Procedures in Videos

“…However, their performance on Charades [79] or AVA [32], shown in Figure 7 (b), is not satisfactory to conduct step-level detection before verification. Although [3,5,11,28,53,54,75,76,92,[96][97][98]100,102,103] perform well on ActivityNet [8] or THUMOS14 [38], it lacks persuasion since the two datasets either contain a few action classes or action instances per video. Thus, we introduce a simple but effective baseline CosAlignment Transformer (abbreviated as CAT), which leverages 2D convolution to extract discriminative features from sampled frames and utilizes a transformer to model inter-step temporal cor-relation in a video clip.…”