Self-Supervised Contrastive Learning of Protein Representations By Mutual Information Maximization

Abstract

Pretrained embedding representations of biological sequences which capture meaningful properties can alleviate many problems associated with supervised learning in biology. We apply the principle of mutual information maximization between local and global information as a self-supervised pretraining signal for protein embeddings. To do so, we divide protein sequences into fixed size fragments, and train an autoregressive model to distinguish between subsequent fragments from the same protein and fragments from random proteins. Our model, CPCProt, achieves comparable performance to state-of-the-art self-supervised models for protein sequence embeddings on various downstream tasks, but reduces the number of parameters down to 0.9% to 8.9% of benchmarked models. Further, we explore how downstream assessment protocols affect embedding evaluation, and the effect of contrastive learning hyperparameters on empirical performance. We hope that these results will inform the development of contrastive learning methods in protein biology and other modalities.

MSA Transformer

“…Alternatives to the masked language modeling objective have also been explored, such as conditional generation (Madani et al, 2020) and contrastive loss functions (Lu et al, 2020). Most relevant to our work, Sturmfels et al (2020) and Sercu et al (2020) study alternative learning objectives using sets of sequences for supervision.…”

Discovering molecular features of intrinsically disordered regions by using evolution for contrastive learning

“…These tasks are not directly useful, but are intended to teach the model transferable skills and representations, and are designed so the models learn autonomously without expert labels. Several self-supervised learning methods have been applied to protein sequences, and have been effective in teaching the models features that are useful for downstream analyses (Alley et al, 2019;Heinzinger et al, 2019;Rao et al, 2019Rao et al, , 2021Lu et al, 2020;Rives et al, 2021). However, the majority of these tasks directly repurpose methods from natural language processing (Alley et al, 2019;Heinzinger et al, 2019;Rao et al, 2019;Rives et al, 2021), and it is unclear what kinds of features the tasks induce the models to learn in the context of protein sequences.…”

Protein Structural Alignments From Sequence

“…Since we have only sequence, we need to capture the structural propensities of a given sequence. Many potential choices for embeddings apropos to structurally informed tasks are computationally expensive [11, 29, 30, 31, 32, 33, 34]. For this paper we have selected a well vetted language model [11] to cast the protein residues to a latent space with the intention of recovering the underlying grammars behind protein sequences.…”

“…Often these models will learn these representations by being trained to predict randomly masked residues within a protein sequence. Multiple studies have shown the merits of these models when performing protein structure prediction, remote homology and protein design [30, 35, 29, 32, 31, 33, 34]. Here, we have used the pretrained LSTM PFam model from [11].…”

“…Recent studies have shown the benefits of using high-capacity self-supervised protein language models to predict protein structure [11, 29, 30, 31, 32, 33, 34]. Pre-training these protein models on large unlabeled datasets can reduce the number of labeled data required to train classifiers, while improving the generalizability.…”