Cross-Modal Discrete Representation Learning

Abstract

In contrast to recent advances focusing on highlevel representation learning across modalities, in this work we present a self-supervised learning framework that is able to learn a representation that captures finer levels of granularity across different modalities such as concepts or events represented by visual objects or spoken words. Our framework relies on a discretized embedding space created via vector quantization that is shared across different modalities. Beyond the shared embedding space, we propose a Cross-Modal Code Matching objective that forces the representations from different views (modalities) to have a similar distribution over the discrete embedding space such that cross-modal objects/actions localization can be performed without direct supervision. We show that the proposed discretized multi-modal finegrained representation (e.g., pixel/word/frame) can complement high-level summary representations (e.g., video/sentence/waveform) for improved performance on cross-modal retrieval tasks. We also observe that the discretized representation uses individual clusters to represent the same semantic concept across modalities.

Video-Text Retrieval by Supervised Multi-Space Multi-Grained Alignment

“…Further, TS2-Net (Liu et al, 2022b) proposes a novel token shift and selection transformer architecture that adjusts the token sequence and selects informative tokens in both temporal and spatial dimensions from input video samples. Later, DiscreteCodebook (Liu et al, 2022a) propose to align modalities in a space filled with concepts, which are randomly initialled and unsupervisedly updated, while VCM propose to construct a space with unsupervisedly clustered visual concepts.…”

“…In this part, we present a general video-text retrieval framework used by previous methods (Luo et al, 2022;Liu et al, 2022a). With this paradigm, we can obtain three representations for different modalities from the original space, i.e., frame representation r f , video representation r v , and sentence representation r s by modality-dependent encoders.…”

“…To show the empirical efficiency of our SUMA, we train models on MSR-VTT (Xu et al, 2016), MSVD (Chen and Dolan, 2011), and Activi-tyNet (Fabian Caba Heilbron and Niebles, 2015). For a fair comparison, we only compare our methods with methods that are based on CLIP (Radford et al, 2021), i.e., Clip4Clip (Luo et al, 2022), CLIP2TV (Gao et al, 2021), X-CLIP , DiscreteCodebook (Liu et al, 2022a), TS2-Net (Liu et al, 2022b), CLIP2Video (Park et al, 2022), VCM , HiSE (Wang et al, 2022a), Align&Tell (Wang et al, 2022b), Center-CLIP (Zhao et al, 2022), and X-Pool (Gorti et al, 2022). Implementation details and evaluation protocols are deferred to the Appendix.…”

“…As a fundamental task in visual-language understanding (Xu et al, 2021;Park et al, 2022;Miyawaki et al, 2022), video-text retrieval (VTR) (Luo et al, 2022;Gao et al, 2021;Liu et al, 2022a;Zhao et al, 2022;Gorti et al, 2022) has attracted interest from academia and industry. Although recent years have witnessed the rapid development of VTR with the support from powerful pretraining models (Luo et al, 2022;Gao et al, 2021;Liu et al, 2022a), improved retrieval methods (Bertasius et al, 2021;Dong et al, 2019;, and videolanguage datasets construction (Xu et al, 2016), it remains challenging to precisely match video and language due to the raw data being in heterogeneous spaces with significant differences.…”

“…However, as the video and text representations often come from modality-independent encoders, it is challenging to directly compare and calculate the similarities between representations of different modalities from different encoders. To alleviate the mismatch caused by heterogeneous encoders and data formats, Liu et al (2022a); proposed to align modalities in a common space filled with learned concepts without supervision. However, because of the unsupervised design, these concepts are either randomly initialized or updated in an unsupervised fashion, which blocks the power of that aligned space.…”