Fast-Slow Transformer for Visually Grounding Speech

Abstract

We present Fast-Slow Transformer for Visually Grounding Speech, or FaST-VGS. FaST-VGS is a Transformer-based model for learning the associations between raw speech waveforms and visual images. The model unifies dual-encoder and cross-attention architectures into a single model, reaping the superior retrieval speed of the former along with the accuracy of the latter. FaST-VGS achieves state-of-the-art speech-image retrieval accuracy on benchmark datasets, and its learned representations exhibit strong performance on the ZeroSpeech 2021 phonetic and semantic tasks.

Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling

“…These works have looked at a variety of tasks, such as speech-image retrieval (Harwath, Torralba, and Glass 2016;Chrupała 2019;Ilharco, Zhang, and Baldridge 2019;Mortazavi 2020;Sanabria, Waters, and Baldridge 2021), automatic speech recognition (Sun, Harwath, and Glass 2016;Palaskar, Sanabria, and Metze 2018;Hsu, Harwath, and Glass 2019), word detection and localization (Kamper et al 2017;Harwath and Glass 2017;Merkx, Frank, and Ernestus 2019;Wang and Hasegawa-Johnson 2020;Olaleye and Kamper 2021), hierarchical linguistic unit analysis (Chrupała, Gelderloos, and Alishahi 2017;Harwath, Hsu, and Glass 2020), cross-modality alignment Wang et al 2021;Khorrami and Räsänen 2021), speech segmentation , speech generation (Hsu et al 2021b), and learning multilingual speech representations (Harwath, Chuang, and Glass 2018;Kamper and Roth 2018;Havard, Chevrot, and Besacier 2020;Ohishi et al 2020). In this paper, we study the recently proposed FaST-VGS (Peng and Harwath 2021) speech-image retrieval model, and and propose a novel extention of the model that incorporates a wav2vec2.0style ) masked language modeling objective in a multi-task learning framework.…”

“…In this section, we describe two models, namely FaST-VGS and FaST-VGS+. FaST-VGS is a transformer-based VGS model proposed by (Peng and Harwath 2021), which is trained using a contrastive speech-image retrieval loss (Ilharco, Zhang, and Baldridge 2019). FaST-VGS+ augments FaST-VGS with a wav2vec2.0-style masked language modeling loss ).…”

“…In particular, FaST-VGS produces two similarity scores for every input pair -the coarse similarity S c , which is calculated by taking the dot product of the output ClS I and ClS A of the two unimodal encoders; and the fine similarity S f , which is calculated by passing the output ClS I and ClS A of the cross-modal encoder to an MLP. As demonstrated by Peng and Harwath (2021), retrieval based on the fine similarity S f is more accurate than retrieval based on the coarse similarity s c , but can be much slower, especially when the search database is large. To combine the advantages of fast retrieval using S c and accurate retrieval using S f , Peng and Harwath (2021) proposes the coarse-to-fine retrieval technique (CTF).…”

“…As demonstrated by Peng and Harwath (2021), retrieval based on the fine similarity S f is more accurate than retrieval based on the coarse similarity s c , but can be much slower, especially when the search database is large. To combine the advantages of fast retrieval using S c and accurate retrieval using S f , Peng and Harwath (2021) proposes the coarse-to-fine retrieval technique (CTF). CTF uses a two-pass strategy by using S c to retrieve a set of K c target items from the search database, and only passes the features of these items and the features of the query to the cross-modal encoder.…”

“…In this work, we explore the benefit of incorporating visual context for self-supervised speech representation learning. In particular, we study a recently proposed model FaST-VGS (Peng and Harwath 2021) which is trained via a cross-modal contrastive loss to match visual images with the speech that describes them. In this paper, we also introduce a novel extension of FaST-VGS which we call FaST-VGS+, that augments FaST-VGS with a wav2vec2.0-type ) masked language modeling loss applied to the speech audio.…”