DEBUG: A Dense Bottom-Up Grounding Approach for Natural Language Video Localization

Abstract

In this paper, we focus on natural language video localization: localizing (i.e., grounding) a natural language description in a long and untrimmed video sequence. All currently published models for addressing this problem can be categorized into two types: (i) top-down approach: it does classification and regression for a set of pre-cut video segment candidates; (ii) bottom-up approach: it directly predicts probabilities for each video frame as the temporal boundaries (i.e., start and end time point). However, both two approaches suffer several limitations: the former is computationintensive for densely placed candidates, while the latter has trailed the performance of the top-down counterpart thus far. To this end, we propose a novel dense bottom-up framework: DEnse Bottom-Up Grounding (DEBUG). DE-BUG regards all frames falling in the ground truth segment as foreground, and each foreground frame regresses the unique distances from its location to bi-directional ground truth boundaries. Extensive experiments on three challenging benchmarks (TACoS, Charades-STA, and ActivityNet Captions) show that DE-BUG is able to match the speed of bottom-up models while surpassing the performance of the state-of-the-art top-down models. * Chujie Lu and Long Chen are co-first authors with equal contributions. † Jun Xiao is the corresponding author.Language Query: People are scrubbing the ice in front of a ball. 89.7s 94.2s

Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language

“…-anchor based methods: TGN [16], CMIN [17] and CBP [38], SCDM [18], -anchor free methods: ACRN [12], ROLE [23], SLTA [28], DEBUG [27], VSLNet [19], GDP [26] LGI [24], ABLR [20], TMLGA [25], ExCL [21] and DRN [22], -reinforcement learning based methods: RWM-RL [29], SM-RL [30], TripNet [31] and TSP-RPL [32],…”

“…In terms of their context modeling, most approaches [12], [16], [17], [20]- [22], [25] gradually aggregate the context information through a recurrent structure. Some approaches [18], [23], [29] model surrounding clips as the local context using 1D convolution layers, while other approaches model the entire clip as the global context through self-attention modules [19], [24], [26], [27]. Since clips are the shortest moments, the clip-level context is a subset of moment-level context.…”

Dual Path Interaction Network for Video Moment Localization

“…Though some of them use an additional regression layer to predict the offsets, their candidate-level feature is not suitable for boundary-level regression and result in inferior performance. On the other hand, by comparing our method with frame-based bottom-up approaches (DEBUG [27], TGN [4], CBP [36], GDP [6]), we can observe that our method works better. Since these approaches only use frame-level representation for moment localization, the boundary features are unaware of the moment content they constitute and lack of consistency, which results in poor performance.…”

“…Following [4], Authors of [5] utilized cross-gated attended recurrent network with the cross-modal interactor and the self interactor to catch the interactions between the sentence and video. Authors of [27] making full use of positive samples to alleviate the severe imbalance problem. Authors of [6] use a Graph-FPN layer to encoder scene relationships and semantics.…”

“…We evaluate our proposed DPIN approach on three datasets and compare our model with the state-of-the-art methods, including: Candidatebased (top-down) approaches: CTRL [9], MCF [39], ACRN [24], SAP [7], CMIN [50], ACL [10], SCDM [43], ROLE [25], SLTA [16],MAN [47], Xu et al [41], SCDM [43], 2D-TAN [48]. Frame-based (Bottomup) approaches: ABLR [44],GDP [6], TGN [4], CBP [36], ExCL [11], DEBUG [27]. Reinforcement learning approach: SM-RL [37].…”