AVLnet: Learning Audio-Visual Language Representations from Instructional Videos

Abstract

Current methods for learning visually grounded language from videos often rely on time-consuming and expensive data collection, such as human annotated textual summaries or machine generated automatic speech recognition transcripts. In this work, we introduce Audio-Video Language Network (AVLnet), a self-supervised network that learns a shared audio-visual embedding space directly from raw video inputs. We circumvent the need for annotation and instead learn audiovisual language representations directly from randomly segmented video clips and their raw audio waveforms. We train AVLnet on publicly available instructional videos and evaluate our model on video clip and language retrieval tasks on three video datasets. Our proposed model outperforms several state-of-the-art text-video baselines by up to 11.8% in a video clip retrieval task, despite operating on the raw audio instead of manually annotated text captions. Further, we show AVLnet is capable of integrating textual information, increasing its modularity and improving performance by up to 20.3% on the video clip retrieval task. Finally, we perform analysis of AVLnet's learned representations, showing our model has learned to relate visual objects with salient words and natural sounds.

Cascaded Multilingual Audio-Visual Learning from Videos

“…We use the following instructional video datasets: HowTo100M [13] (1.2M videos), YouCook2 [14] and YouCook-Japanese. For YouCook2, we use 9,586 train clips and 3,350 validation clips as in [12]. We evaluate performance on audio to video clip retrieval and video clip to audio retrieval using the standard recall metrics R@1, R@5, R@10.…”

“…Boggust et al [31] applied an image-caption [4] model to videos, using a single image frame from entire video clips to perform video to audio retrieval. Rouditchenko et al [12] proposed the AVLnet model that pools visual information over entire clips using 2D and 3D visual CNNs. We use AVLnet trained on English HowTo100M videos and apply it to cooking videos in Japanese and images and spoken captions in Japanese and Hindi.…”

“…Our goal is to learn audio-visual representations for videos in languages other than English using AVLnet [12]. AVLnet is trained through a contrastive loss to discriminate between temporally aligned audio-video pairs and temporally mismatched pairs from both within the same video and from other videos.…”

MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound

“…Models like CLIP learn image classification by matching images with their captions, contrastively [130,91,62]. Recent work has explored this paradigm for matching video frames with their transcripts [121], with their audio signal [96,114], or both [3,2]; these works likewise perform well on single-modality tasks like audio classification and activity recognition. These independent encoders can be combined through late fusion [96], yet late fusion is strictly less expressive than our proposed joint encoding (early fusion) approach.…”

Video-Text Pre-training with Learned Regions

“…Though some of these methods adopt expert features including object [79], sound [19,58] and speech [19] information. Compared with the most related work, Frozen [6], our RegionLearner brings significant improvements on text-tovideo retrieval.…”