Bridging Video-text Retrieval with Multiple Choice Questions

Ge, Yuying; Ge, Yixiao; Liu, Xihui; Li, Dian; Shan, Ying; Qie, Xiaohu; Luo, Ping

doi:10.1109/cvpr52688.2022.01569

Cited by 83 publications

(28 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Existing works either learn unimodal encoders by distinguishing the positive pair(s) from the unpaired samples [3,28,41] or focus on one multimodal encoder for joint feature learning with masked image/language modeling and image-text matching losses [12,27,29]. Additionally, some approaches seek fine-grained supervision for cross-modal interaction [20,30,[54][55][56]. For example, GLIP [30] proposed to align the bounding boxes with corresponding phrases in the text.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervision

Xu¹,

Hou²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

In this paper, we consider the problem of openvocabulary semantic segmentation (OVS), which aims to segment objects of arbitrary classes instead of pre-defined, closed-set categories. The main contributions are as follows: First, we propose a transformer-based model for OVS, termed as OVSegmentor, which only exploits webcrawled image-text pairs for pre-training without using any mask annotations. OVSegmentor assembles the image pixels into a set of learnable group tokens via a slot-attention based binding module, and aligns the group tokens to the corresponding caption embedding. Second, we propose two proxy tasks for training, namely masked entity completion and cross-image mask consistency. The former aims to infer all masked entities in the caption given the group tokens, that enables the model to learn fine-grained alignment between visual groups and text entities. The latter enforces consistent mask predictions between images that contain shared entities, which encourages the model to learn visual invariance. Third, we construct CC4M dataset for pre-training by filtering CC12M with frequently appeared entities, which significantly improves training efficiency. Fourth, we perform zero-shot transfer on three benchmark datasets, PASCAL VOC 2012, PASCAL Context, and COCO Object. Our model achieves superior segmentation results over the state-of-the-art method by using only 3% data (4M vs 134M) for pre-training. Code and pre-trained models will be released for future research.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Entities vs nouns: masking and predicting noun phrases in the sentence [20] is one feasible option to learn fine-grained vision-text matching. However, noun masking leads to 3.5% lower mIoU than our entity masking strategy.…”

Section: Ablation Studymentioning

confidence: 99%

Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervision

Xu¹,

Hou²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Popular image-language models such as CLIP [83] and ALIGN [48] are trained on massive datasets by using web images and alt-text. Similarly, videolanguage models are catching up and can be categorised into two broad directions: (i) adapting image-language models for videos [8,22,49,50,62,65,71,108,110,119], and (ii) pure video-based models that are learned using large video-text datasets [3,7,[26][27][28]30,57,61,64,67,68,95,117]. Recently, a new paradigm of post-pretraining has emerged where an existing image-or video-language model goes through another stage of self-supervised pretraining on a small amount of video data before it is evaluated on downstream tasks [65,119].…”

Section: Foundational Video-language Modelsmentioning

confidence: 99%

Test of Time: Instilling Video-Language Models with a Sense of Time

Bagad¹,

Tapaswi²,

Snoek³

2023

Preprint

View full text Add to dashboard Cite

Modeling and understanding time remains a challenge in contemporary video understanding models. With language emerging as a key driver towards powerful generalization, it is imperative for foundational video-language models to have a sense of time. In this paper, we consider a specific aspect of temporal understanding: consistency of time order as elicited by before/after relations. We establish that six existing video-language models struggle to understand even such simple temporal relations. We then question whether it is feasible to equip these foundational models with temporal awareness without re-training them from scratch. Towards this, we propose a temporal adaptation recipe on top of one such model, VideoCLIP, based on post-pretraining on a small amount of video-text data. We conduct a zero-shot evaluation of the adapted models on six datasets for three downstream tasks which require a varying degree of time awareness. We observe encouraging performance gains especially when the task needs higher time awareness. Our work serves as a first step towards probing and instilling a sense of time in existing video-language models without the need for data and compute-intense training from scratch.

show abstract

“…During training, the PEM task asks the model to predict the entity pseudo-labels (i.e., normalized similarity scores) for randomly-selected video crops. In BridgeFormer (Ge et al, 2022), the authors exploit the rich semantics of text (i.e., nouns and verbs) to build question-answer pairs to form a question answering task as a pretext task, with which the model can be trained to capture more re-gional content and temporal dynamics. Wang et al (2022c) proposes an object-aware Transformer to leverage bounding boxes and object tags to guide the training process.…”

Section: Advanced Pre-training Tasksmentioning

confidence: 99%

Vision-Language Pre-training: Basics, Recent Advances, and Future Trends

Gan¹,

Fu²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper surveys vision-language pre-training (VLP) methods for multimodal intelligence that have been developed in the last few years. We group these approaches into three categories: (i) VLP for image-text tasks, such as image captioning, image-text retrieval, visual question answering, and visual grounding; (ii) VLP for core computer vision tasks, such as (open-set) image classification, object detection, and segmentation; and (iii) VLP for video-text tasks, such as video captioning, video-text retrieval, and video question answering. For each category, we present a comprehensive review of state-of-the-art methods, and discuss the progress that has been made and challenges still being faced, using specific systems and models as case studies. In addition, for each category, we discuss advanced topics being actively explored in the research community, such as big foundation models, unified modeling, in-context few-shot learning, knowledge, robustness, and computer vision in the wild, to name a few.♠ Zhe Gan and Jianfeng Gao initiated the project. Zhe Gan and Linjie Li took lead in the writing of Chapter 1. Linjie Li and Jianfeng Gao took lead in the writing of Chapter 2. Zhe Gan further took lead in the writing of Chapter 3 and 7. Chunyuan Li took lead in the writing of Chapter 4. Linjie Li further took lead in the writing of Chapter 5. Lijuan Wang and Zicheng Liu took lead in the writing of Chapter 6. All the authors provided project advice, and contributed to paper editing and proofreading.

show abstract

Bridging Video-text Retrieval with Multiple Choice Questions

Cited by 83 publications

References 24 publications

Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervision

Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervision

Test of Time: Instilling Video-Language Models with a Sense of Time

Vision-Language Pre-training: Basics, Recent Advances, and Future Trends

Contact Info

Product

Resources

About