Fine-grained Multi-Modal Self-Supervised Learning

Abstract

Multi-Modal Self-Supervised Learning from videos has been shown to improve model's performance on various downstream tasks. However, such Self-Supervised pretraining requires large batch sizes and a large amount of computation resources due to the noise present in the uncurated data. This is partly due to the fact that the prevalent training scheme is trained on coarse-grained setting, in which vectors representing the whole video clips or natural language sentences are used for computing similarity. Such scheme makes training noisy as part of the video clips can be totally not correlated with the othermodality input such as text description. In this paper, we propose a fine-grained multimodal self-supervised training scheme that computes the similarity between embeddings at finer-scale (such as individual feature map embeddings and embeddings of phrases), and uses attention mechanisms to reduce noisy pairs' weighting in the loss function. We show that with the proposed pre-training scheme, we can train smaller models, with smaller batch-size and much less computational resources to achieve downstream tasks performances comparable to State-Of-The-Art, for tasks including action recognition and text-image retrievals.

Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learning

“…It is suboptimal for generative tasks or dense prediction tasks. In computer vision, some studies perform contrastive learning at pixel-level to learn finer input representations [40]- [42]. In the ST task, we need to conduct fine granularity contrastive at frame-level so that each speech frame in the encoder has precise semantics.…”