Dual Path Interaction Network for Video Moment Localization

Abstract

Video moment localization aims to localize a specific moment in a video by a natural language query. Previous works either use alignment information to find out the best-matching candidate (i.e., topdown approach) or use discrimination information to predict the temporal boundaries of the match (i.e., bottom-up approach). Little research has taken both the candidate-level alignment information and frame-level boundary information together and considers the complementarity between them. In this paper, we propose a unified top-down and bottom-up approach called Dual Path Interaction Network (DPIN), where the alignment and discrimination information are closely connected to jointly make the prediction. Our model includes a boundary prediction pathway encoding the frame-level representation and an alignment pathway extracting the candidatelevel representation. The two branches of our network predict two complementary but different representations for moment localization. To enforce the consistency and strengthen the connection between the two representations, we propose a semantically conditioned interaction module. The experimental results on three popular benchmarks (i.e., TACoS, Charades-STA, and Activity-Caption) demonstrate that the proposed approach effectively localizes the relevant moment and outperforms the state-of-the-art approaches. CCS CONCEPTS • Information systems → Video search; Novelty in information retrieval.

A Survey on Natural Language Video Localization

“…However, it is also a highly computationally intensive scheme, so they use some tricks such as sampling to reduce the amount of operations. Wang et al [53] argued that utilizing both the frame-level and the candidate-level features will create complementary advantages. Therefore, they proposed Dual Path Interaction Network (DPIN), which utilizes Semantically Conditioned Interaction to accomplish the information transformation between the two levels.…”

Explore-And-Match: Bridging Proposal-Based and Proposal-Free With Transformer for Sentence Grounding in Videos

“…While proposal-based approaches show reliable results, they are sensitive to proposal quality and suffer from the prohibitive cost of creating proposals, as well as the computationally inefficient comparison of all proposaltarget pairings. Another line of works are the proposal-free approaches [6,7,8,15,29,31,36,42,45,52,53,55], which tries to regress the timespans directly. They are more flexible than proposal-based approaches in terms of granularity.…”

“…The majority of existing methods for video grounding can be categorized into two families: 1) proposal-based methods [2,5,13,14,17,25,26,33,43,45,48,50,54,56,57,58], which All codes and models will be made available shortly. generate a bunch of proposals in advance and select the best match with target spans, and 2) proposal-free methods [6,7,8,15,29,31,36,42,45,52,53,55], which estimate start and end timestamps aligned to the given description directly. The proposal-based approaches generally show strong performance at the expense of prohibitive cost of proposal generation.…”

Cross-Sentence Temporal and Semantic Relations in Video Activity Localisation

“…The latest methods either follow SAP to predict the probabilities of boundary across frames [3,32,6,4,22] or in the same spirit as MCN to select from a set of pre-defined proposals constructed by explicit sliding windows [9,19] or implicit multi-granularity anchors [33,29,31]. Recently, DPIN [27] proposed to combine the two localisation strategies by a dual path interaction network so to take the advantage of both. Regardless of their remarkable success, fully-supervised methods rely heavily on the fine-grained temporal annotation, which is not only expensive but also prone to subjective bias [1].…”