Adversarial Video Moment Retrieval by Jointly Modeling Ranking and Localization

Abstract

Retrieving video moments from an untrimmed video given a natural language as the query is a challenging task in both academia and industry. Although much effort has been made to address this issue, traditional video moment ranking methods are unable to generate reasonable video moment candidates and video moment localization approaches are not applicable to large-scale retrieval scenario. How to combine ranking and localization into a unified framework to overcome their drawbacks and reinforce each other is rarely considered. Toward this end, we contribute a novel solution to thoroughly investigate the video moment retrieval issue under the adversarial learning paradigm. The key of our solution is to formulate the video moment retrieval task as an adversarial learning problem with two tightly connected components. Specifically, a reinforcement learning is employed as a generator to produce a set of possible video moments. Meanwhile, a pairwise ranking model is utilized as a discriminator to rank the generated video moments and the ground truth. Finally, the generator and the discriminator are mutually reinforced in the adversarial learning framework, which is able to jointly optimize the performance of both video moment ranking and video moment localization. Extensive experiments on two well-known datasets have well verified the effectiveness and rationality of our proposed solution. CCS CONCEPTS • Information systems → Multimedia and multimodal retrieval; • Theory of computation → Adversarial learning; Reinforcement learning.

The Elements of Temporal Sentence Grounding in Videos: A Survey and Future Directions

“…21). AVMR [129] treats RL-based module as a generator, and devises a Bayesian ranking module as discriminator to rank proposals. STRONG [130] considers both appearance and motion features, and employs parallel spatial-level and temporal-level RL modules for moment localization.…”

A Survey on Natural Language Video Localization

“…To align the text and video, Hahn et al [68] used gated attention [69] showing their model achieves faster localization than the traditional one. Cao et al [70] proposed an adversarial learning paradigm combining reinforcement learning and moment ranking to solve the NLVL problem. In this way, it is more easier to learn the differences between moments within the same video and can select target segment more efficiently.…”

A Survey on Temporal Sentence Grounding in Videos

“…The action space for each step is a set of handcraft-designed temporal transformations (e.g., shifting, scaling). The typical methods include R-W-M [22], SM-RL [62], TripNet [21], STRONG [2], TSP-PRL [65] and AVMR [3].…”

“…The alignment network will predict a confidence score to determine when to stop. Meanwhile, AVMR [3] addresses TSGV under the adversarial learning paradigm, which designs a RL-based proposal generator to generate proposal candidates and employs Bayesian Personalized Ranking as a discriminator to rank these generated moment proposals in a pairwise manner.…”