Context-aware Biaffine Localizing Network for Temporal Sentence Grounding

Abstract

This paper addresses the problem of temporal sentence grounding (TSG), which aims to identify the temporal boundary of a specific segment from an untrimmed video by a sentence query. Previous works either compare pre-defined candidate segments with the query and select the best one by ranking, or directly regress the boundary timestamps of the target segment. In this paper, we propose a novel localization framework that scores all pairs of start and end indices within the video simultaneously with a biaffine mechanism. In particular, we present a Contextaware Biaffine Localizing Network (CBLN) which incorporates both local and global contexts into features of each start/end position for biaffine-based localization. The local contexts from the adjacent frames help distinguish the visually similar appearance, and the global contexts from the entire video contribute to reasoning the temporal relation. Besides, we also develop a multi-modal self-attention module to provide fine-grained query-guided video representation for this biaffine strategy. Extensive experiments show that our CBLN significantly outperforms state-of-thearts on three public datasets (ActivityNet Captions, TACoS, and Charades-STA), demonstrating the effectiveness of the proposed localization framework. The code is available at https://github.com/liudaizong/CBLN.

The Elements of Temporal Sentence Grounding in Videos: A Survey and Future Directions

“…DORi [119] incorporates appearance features and captures the relations between objects and actions guided by query. CBLN [120] addresses TSGV from a new perspective. It reformulates TSGV by scoring all pairs of start and end indices simultaneously and predict moment with a biaffine structure.…”

Exploring Optical-Flow-Guided Motion and Detection-Based Appearance for Temporal Sentence Grounding

“…2) Query encoder Following previous works [16], [20], [78], we first employ the Glove model [79] to embed each word of the given sentence query into a dense vector. Then, we use multi-head self-attention [80] and Bi-GRU [81] modules to encode its sequential information.…”

“…At last, we apply grounding heads on the feature H to predict the target segment semantically corresponding to the query information. There are many grounding heads proposed in recent years: proposal-ranking based grounding head [18], [20], [71], [78] and the boundary-regression grounding head [25]- [27]. In this paper, we follow the former one [18], [20], [74] to determine the target video segment with pre-defined segment proposals.…”

“…(1) Proposal-based approach: CTRL [16], ACRN [67], QSPN [68], SCDM [74], BPNet [88], CMIN [18], 2DTAN [21], DRN [89], FIAN [90], CBLN [78]; (2) Proposal-free approach: CBP [72], GDP [25], LGI [27], VSLNet [28], IVG-DCL [91], ACRM [2]; (3) Detection-based approach: MMRG [29].…”

MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptions

“…State-of-the-art video grounding methods [11,12,14,25,32,34,35] have relied on existing benchmarks to design novel modules (e.g. proposal generation, context modeling, and multi-modality fusion).…”