Span-based Localizing Network for Natural Language Video Localization

Abstract

Given an untrimmed video and a text query, natural language video localization (NLVL) is to locate a matching span from the video that semantically corresponds to the query. Existing solutions formulate NLVL either as a ranking task and apply multimodal matching architecture, or as a regression task to directly regress the target video span. In this work, we address NLVL task with a span-based QA approach by treating the input video as text passage. We propose a video span localizing network (VSLNet), on top of the standard span-based QA framework, to address NLVL. The proposed VSLNet tackles the differences between NLVL and span-based QA through a simple and yet effective query-guided highlighting (QGH) strategy. The QGH guides VSLNet to search for matching video span within a highlighted region. Through extensive experiments on three benchmark datasets, we show that the proposed VSLNet outperforms the state-of-the-art methods; and adopting span-based QA framework is a promising direction to solve NLVL. 1 * Corresponding author. 1 https://github.com/IsaacChanghau/VSLNet Language Query: Men are celebrating and an old man gives a trophy to a young boy.

Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language

“…It is worth noting that on TACoS dataset (see Table 4), our MS-2D-TAN surpasses the previous best approach CBP [38] , by approximate 18 points and 25 points in term of Rank1@0.3 and Rank5@0.3, respectively. Moreover, on the large-scale ActivityNet Captions dataset, MS-2D-TAN also outperforms the top ranked method DRN [22] and VSLNet [19] with respect to IoU @0.5 and 0.7. It validates that MS-2D-TAN is able to localize the moment boundary more precisely.…”

“…-anchor based methods: TGN [16], CMIN [17] and CBP [38], SCDM [18], -anchor free methods: ACRN [12], ROLE [23], SLTA [28], DEBUG [27], VSLNet [19], GDP [26] LGI [24], ABLR [20], TMLGA [25], ExCL [21] and DRN [22], -reinforcement learning based methods: RWM-RL [29], SM-RL [30], TripNet [31] and TSP-RPL [32],…”

“…In general, there are three common ways to map a clip to a moment score: anchor-based methods, anchor-free methods, and RL-based methods. Anchor-based methods define a set of anchors with a fixed length for each clip [16]- [18], while anchor-free methods directly predict the start and the end time through classification [19], [20] or regression [12], [21]- [28]. RL-based methods model the task as a sequential decision-making problem and solve it by reinforcement learning [29]- [32].…”

Diving Into The Relations: Leveraging Semantic and Visual Structures For Video Moment Retrieval

“…It has various applications such as robotic navigation, video entertainment, and autonomous driving, to *Shucheng Huang(schuang@just.edu.cn) is the corresponding author. name a few [1,2,3,4,5]. Despite much progress has been achieved in recent years [6,7,8,9,10,11,12,13], VMR remains difficult due to the harsh nature of videos and texts, including complex temporal relations, fine-grained semantic structures, and huge cross-modal gap between visual and textual features [11,14,15,16].…”

“…The current dominant approaches for video moment retrieval is to learn the semantic correlation between the query and the video. To this end, numerous cross-modality alignment strategies are designed such as cross-attention [1,2], recurrent neural networks [17,18], semantic conditioned dynamic modulation [11], and 2D temporal adjacent network [14]. Although achieving favorable performance, most current methods do not take full advantage of the fine-grained and comprehensive relation information in both semantic and visual structures: (1) Many existing VMR approaches only encode the semantic information of the query in a global manner [9,19,10,20,14,12,13], i.e., embedding the texts into a global vector representation by using LSTM or other sequential models, but ignore the intrinsic and fine-grained structure of the sentence.…”