AdaFocus V2: End-to-End Training of Spatial Dynamic Networks for Video Recognition

ViGAT: Bottom-Up Event Recognition and Explanation in Video Using Factorized Graph Attention Network

“…ii) Concerning our ViGAT variant that utilizes a ResNet backbone pretrained on ImageNet, this outperforms the bestperforming literature approaches that similarly use a ResNet backbone in FCVID and ActivityNet (see Tables 1 and 3). Specifically, we observe a significant performance gain of 1% over AdaFocusV2 [19], which is the previous state-of-theart method. We also see that ViGAT provides a performance improvement of 1.4% over ObjectGraphs [5], which is the best previous bottom-up method.…”

“…The proposed approach is compared against the top-scoring approaches of the literature on the three employed datasets, specifically, TBN [44], BAT [16], MARS [62], Fast-S3D [38], RMS [64], CGNL [30], ATFR [72], Ada3D [17], TCPNet [45], LgNet [68], ST-VLAD [50], PivotCorrNN [53], LiteEval [57], AdaFrame [54], Listen to Look [56], SCSampler [73], AR-Net [7], SMART [59], ObjectGraphs [5], MARL [55], FrameExit [6] and AdaFocusV2 [19] (note that not all of these works report results for all the datasets mAP(%) AdaFrame [54] 71.5 Listen to Look [56] 72.3 LiteEval [57] 72.7 SCSampler [73] 72.9 AR-Net [7] 73.8 FrameExit [6] 77.3 AdaFocusV2 [19] 79.0 AR-Net (EfficientNet backbone) [7] 79.7 MARL (ResNet backbone on Kinetics) [55] 82.9 FrameExit (X3D-S backbone) [6] 87 used in the present work). The reported results on FCVID, MiniKinetics and ActivityNet are shown in Tables 1, 2 and 3, respectively.…”

“…Contrarily to the above, AdaFocus [58] utilizes a reinforcement learning policy network to leverage spatial redundancy, i.e., selects the most salient regions in the video frames with respect to the action recognition task. In [19], AdaFocusV2 extends [58] by replacing reinforcement learning with a differentiable interpolation-based patch selection operation, enabling efficient end-to-end optimization. The above methods operate on untrimmed videos (i.e., videos that contain many irrelevant frames to the underlying action), where it is much easier to identify and discard less-significant image regions or entire frames.…”

“…Higher IC and lower AD indicate a better explanation. Additionally, we utilize two more general explainability measures, fidelity minus (F −) and fidelity plus (F +) [86], defined as mAP(%) ST-VLAD [50] 77.5 PivotCorrNN [53] 77.6 LiteEval [57] 80.0 AdaFrame [54] 80.2 SCSampler [73] 81.0 AR-Net [7] 81.3 SMART [59] 82.1 AR-Net (EfficientNet backbone) [7] 84.4 ObjectGraphs [5] 84.6 AdaFocusV2 [19] 85.0 ViGAT (proposed; ResNet backbone) 86.0 ViGAT (proposed; ViT backbone) 88.1…”

“…Such an approach can also facilitate the generation of object-and frame-based explanations about the event recognition outcome. An example of this is shown in the second row of the figure . a large amount of them is irrelevant and does not need to be thoroughly analyzed [15]- [19].…”