Fine-grained Iterative Attention Network for Temporal Language Localization in Videos

Abstract

Temporal language localization in videos aims to ground one video segment in an untrimmed video based on a given sentence query. To tackle this task, designing an effective model to extract grounding information from both visual and textual modalities is crucial. However, most previous attempts in this field only focus on unidirectional interactions from video to query, which emphasizes which words to listen and attends to sentence information via vanilla soft attention, but clues from query-by-video interactions implying where to look are not taken into consideration. In this paper, we propose a Fine-grained Iterative Attention Network (FIAN) that consists of an iterative attention module for bilateral query-video in-formation extraction. Specifically, in the iterative attention module, each word in the query is first enhanced by attending to each frame in the video through fine-grained attention, then video iteratively attends to the integrated query. Finally, both video and query information is utilized to provide robust cross-modal representation for further moment localization. In addition, to better predict the target segment, we propose a content-oriented localization strategy instead of applying recent anchor-based localization. We evaluate

Local-enhanced Interaction for Temporal Moment Localization

“…This section will compare our method with several state of the art methods. Since our model belongs to the one-stage methods, we mainly compare it with one-stage methods, which are ABLR [32], ExCL [9], DEBUG [18], TMLGA [24], HVTG [5], VSLnet [34], GDP [4], DRN [33], FIAN [22] and VLG-Net [21]. To further illustrate the effect, we also give the score of some two-stages methods including CTRL [8], SLTA [12], ACRN [16], CBP [26] and 2D-TAN [35].…”

“…The following works [12,15,22] mainly focus on constructing a better interaction model for candidates and query sentence. Jiang et al take advantage of the object-level feature to mine specific details in videos.…”

“…Current methods for this task can be grouped into two categories: two-stages and one-stage. Most of two-stages methods [12,22] adopt the proposal-and-rank pipeline, which first generates many multi-scale candidate moments and then ranks them according to their relevance scores to the query language. Although obtaining huge success, the dense sampling strategy leads to high computation burden, especially for very long videos.…”

Coarse-to-Fine Spatial-Temporal Relationship Inference for Temporal Sentence Grounding

“…Zhang et al [9] first explored the fine-grained semantic information in both videos and sentences and then captured the multi-stage cross-modal interactions. Qu et al [28] proposed the iterative attention module to excavate the grounding clues from both visual and textual modalities. Liu et al [29] reformulated this work as an iterative message passing process over a joint graph that consists of the crossmodal and self-modal relation graphs.…”

“…• FIAN [28]: The FIAN method proposes the iterative attention module, where the visual and textual features reinforce each other to generate robust sentence-aware video representation. Table 1 and 2 report the quantitative performance comparison results on ActivityNet Caption and TACoS datasets, respectively.…”