Progressive Localization Networks for Language-Based Moment Localization

Abstract

This paper targets the task of language-based video moment localization. The language-based setting of this task allows for an open set of target activities, resulting in a large variation of the temporal lengths of video moments. Most existing methods prefer to first sample sufficient candidate moments with various temporal lengths, and then match them with the given query to determine the target moment. However, candidate moments generated with a fixed temporal granularity may be suboptimal to handle the large variation in moment lengths. To this end, we propose a novel multi-stage Progressive Localization Network (PLN) which progressively localizes the target moment in a coarse-to-fine manner. Specifically, each stage of PLN has a localization branch, and focuses on candidate moments that are generated with a specific temporal granularity. The temporal granularities of candidate moments are different across the stages. Moreover, we devise a conditional feature manipulation module and an upsampling connection to bridge the multiple localization branches. In this fashion, the later stages are able to absorb the previously learned information, thus facilitating the more fine-grained localization. Extensive experiments on three public datasets demonstrate the effectiveness of our proposed PLN for language-based moment localization, especially for localizing short moments in long videos.

CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfer

“…Cross-lingual cross-modal retrieval has been garnering increased attention amongst researchers, as it enables the acquisition of images or videos utilizing a non-English query, without relying on human-labeled vision-target language data (Li et al 2023;Wu et al 2023). This method mitigates the constraints of conventional cross-modal retrieval tasks (Dong et al 2022a;Zheng et al 2023) centered on English, and offers a highly efficient and cost-effective solution for target-language based retrieval, greatly reducing the need for human-labeled data. In terms of the model architecture, there are mainly two broad directions to conduct CCR.…”

Unsupervised Domain Adaptative Temporal Sentence Localization with Mutual Information Maximization

“…Therefore, the main challenge in such setting is how to align multi-modal features well to predict precise boundary. Some works (Gao et al 2017;Zhang et al 2019;Yuan et al 2019;Zhang et al 2020b;Chen et al 2018;Qu et al 2020;Fang et al 2022Fang et al , 2023cFang et al , 2020Fang and Hu 2020;Fang et al 2021a,b;Liu et al 2022dLiu et al , 2023bFang et al 2023b;Zheng et al 2023;Zhu et al 2023;Liu et al 2023c,d) integrate sentence information with each fine-grained video clip unit, and predict the scores of candidate segments by gradually merging the fusion feature sequence over time. Without using proposals, some latest methods (Nan et al 2021;Zhang et al 2020a;Chen et al 2020a) are proposed to leverage the interaction between video and sentence to directly predict the starting and ending frames.…”

Fewer Steps, Better Performance: Efficient Cross-Modal Clip Trimming for Video Moment Retrieval Using Language

“…In practice, VMR is an extremely challenging task because the desired model should (i) cover various moment lengths in multiple scenarios; (ii) bridge the semantic gap between different modalities (video and query); (iii) understand the semantic details of different modalities to extract modal-invariant features for optimal retrieval. Most previous VMR works (Zheng et al 2023;Shen et al 2023;Yang et al 2022;Dong et al 2022aDong et al ,b,c, 2023bSun et al 2023;Ma et al 2020;Liu et al 2018Ge et al 2019;Zhang et al 2019a;Qu et al 2023Qu et al , 2021Wen et al 2023bWen et al , 2021Wen et al , 2023a are under fully-supervised setting, where each frame is manually labeled as queryrelevant or not. Therefore, the main challenge in such a setting is how to align multi-modal features well to predict precise moment boundaries.…”