Self-Supervised Learning of Class Embeddings from Video

Abstract

This work explores how to use self-supervised learning on videos to learn a class-specific image embedding that encodes pose and shape information. At train time, two frames of the same video of an object class (e.g. human upper body) are extracted and each encoded to an embedding. Conditioned on these embeddings, the decoder network is tasked to transform one frame into another. To successfully perform long range transformations (e.g. a wrist lowered in one image should be mapped to the same wrist raised in another), we introduce a hierarchical probabilistic network decoder model. Once trained, the embedding can be used for a variety of downstream tasks and domains. We demonstrate our approach quantitatively on three distinct deformable object classeshuman full bodies, upper bodies, faces -and show experimentally that the learned embeddings do indeed generalise. They achieve state-ofthe-art performance in comparison to other self-supervised methods trained on the same datasets, and approach the performance of fully supervised methods.

Multimodal Emotion Recognition With Transformer-Based Self Supervised Feature Fusion

“…We used the pre-trained Fabnet [16] model to obtain embeddings for each frame in the video that contained the speaker's face. The pretext task of Fabnet is specially designed to encourage the network to learn the facial attributes that encodes the landmarks, pose, and emotions.…”

“…Both pre-trained models of RoBERTa [19] and Wav2Vec [17] were accessed from Fairseq code-base [34] and used to extract text and speech SSL features. To download the pre-trained Fabnet model and extract features for video modality, we referred to their publication [16]. To extract features from videos, we cropped faces from each video frame using Retina-Face [42] facial recognition model.…”

“…Considering these problems, we propose a reliable and effective SSL feature fusion mechanism based on the core concepts of Self-Attention [18] and Transformers [18]- [20]. We used three publicly available pre-trained SSL models of RoBERTa [19] to represent text, Wav2Vec [17] to represent speech and FAb-net [16] to represent facial expressions.…”

Self-Supervised Learning of Interpretable Keypoints From Unlabelled Videos

“…Other methods use no supervision, and some no data-driven prior either. The works of [28,45,50,67] learn to match pairs of images of an object, but they do not learn geometric invariants such as keypoints. [54,55,56] do learn sparse and dense landmarks, also without any annotation.…”

“…In this way, our method outputs poses that are directly interpretable. By contrast, state-of-the-art self-supervised keypoint detectors [25,50,54,67,74] do not learn "semantic" keypoints and, in post-processing, they need at least some paired supervision to output human-interpretable keypoints. We highlight this difference in fig.…”