SaProt: Protein Language Modeling with Structure-aware Vocabulary

Su, Jin; Han, Chenchen; Zhou, Yuyang; Shan, Junjie; Zhou, Xibin; Yuan, Fajie

doi:10.1101/2023.10.01.560349

Cited by 43 publications

(39 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Many pretraining tasks still aim to reconstruct natural sequences (He et al, 2021; Notin et al, 2022; Tan et al, 2023; Ma et al, 2023) and so are also likely to primarily learn coevolutionary patterns. Other tasks use structure as an additional input or target, but they generally make only modest improvements on function prediction tasks (Mansoor et al, 2021; Wang et al, 2022; Yang et al, 2023; Su et al, 2023). Supporting the assertion that learning to predict structure may not improve function prediction, Hu et al (2022) show that transfer learning using the AlphaFold2 (Jumper et al, 2021) structure module is less effective for function prediction than transferring PLMs.…”

Section: Discussionmentioning

confidence: 99%

Feature Reuse and Scaling: Understanding Transfer Learning with Protein Language Models

Li,

Amini,

Yue

et al. 2024

Preprint

View full text Add to dashboard Cite

Large pretrained protein language models (PLMs) have improved protein property and structure prediction from sequences via transfer learning, in which weights and representations from PLMs are repurposed for downstream tasks. Although PLMs have shown great promise, currently there is little understanding of how the features learned by pretraining relate to and are useful for downstream tasks. We perform a systematic analysis of transfer learning using PLMs, conducting 370 experiments across a comprehensive suite of factors including different downstream tasks, architectures, model sizes, model depths, and pretraining time. We observe that while almost all downstream tasks do benefit from pretrained models compared to naive sequence representations, for the majority of tasks performance does not scale with pretraining, and instead relies on low-level features learned early in pretraining. Our results point to a mismatch between current PLM pretraining paradigms and most applications of these models, indicating a need for better pretraining methods.

show abstract

Section: Discussionmentioning

confidence: 99%

Feature Reuse and Scaling: Understanding Transfer Learning with Protein Language Models

Li,

Amini,

Yue

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…After the transformative progress brought by deep learning to protein structure prediction [2][3][4][5], predicting protein complex structure and ligand binding sites is fast advancing with AFM and related methods, but also with other deep learning models based on structural representations [73][74][75][76]. Combining the latter [77,78] and, more generally, structural information [79] with the power of sequence-based language models is starting to bring even further progress.…”

Section: Discussionmentioning

confidence: 99%

Pairing interacting protein sequences using masked language modeling

Lupo

Sgarbossa

Bitbol

2023

Preprint

View full text Add to dashboard Cite

Predicting which proteins interact together from amino-acid sequences is an important task. We develop a method to pair interacting protein sequences which leverages the power of protein language models trained on multiple sequence alignments, such as MSA Transformer and the EvoFormer module of AlphaFold. We formulate the problem of pairing interacting partners among the paralogs of two protein families in a differentiable way. We introduce a method called DiffPALM that solves it by exploiting the ability of MSA Transformer to fill in masked amino acids in multiple sequence alignments using the surrounding context. MSA Transformer encodes coevolution between functionally or structurally coupled amino acids. We show that it captures inter-chain coevolution, while it was trained on single-chain data, which means that it can be used out-of-distribution. Relying on MSA Transformer without fine-tuning, DiffPALM outperforms existing coevolution-based pairing methods on difficult benchmarks of shallow multiple sequence alignments extracted from ubiquitous prokaryotic protein datasets. It also outperforms an alternative method based on a state-of-the-art protein language model trained on single sequences. Paired alignments of interacting protein sequences are a crucial ingredient of supervised deep learning methods to predict the three-dimensional structure of protein complexes. DiffPALM substantially improves the structure prediction of some eukaryotic protein complexes by AlphaFold-Multimer, without significantly deteriorating any of those we tested. It also achieves competitive performance with using orthology-based pairing.

show abstract

“…Deep learning tools trained on structural information are likely superior predictors in tasks where spatial features are pre-eminent, as in the case of protein-protein interactions (Wang et al 2022, Heinzinger et al 2023, Shu et al 2023, Su et al 2023). AlphaFold2 and ProteinMPNN, a model developed around a message-passing neural network, have shown potential in predicting fitness variations upon residue substitution (Dauparas et al 2022, Brown et al 2023, Reeves and Kalyaanamoorthy 2023).…”

Section: Discussionmentioning

confidence: 99%

ESM-Scan - a tool to guide amino acid substitutions

Totaro,

Vide,

Zausinger

et al. 2023

Preprint

View full text Add to dashboard Cite

Protein structure prediction and (re)design have gone through a revolution in the last three years. The tremendous progress in these fields has been almost exclusively driven by readily available machine-learning algorithms applied to protein folding and sequence design problems. Despite these advancements, predicting site-specific mutational effects on protein stability and function remains an unsolved problem. This is a persistent challenge mainly because the free energy of large systems is very difficult to compute with absolute accuracy and subtle changes to protein structures are also hard to capture with computational models. Here, we describe the implementation and use of ESM-Scan, which uses the ESM zero-shot predictor to scan entire protein sequences for preferential amino acid changes, thus enabling in-silico deep mutational scanning experiments. We benchmark ESM-Scan on its predictive capabilities for stability and functionality of sequence changes using three publicly available datasets and proceed by experimentally evaluating the tool’s performance on a challenging test case of a blue-light-activated diguanylate cyclase from Methylotenera species (MsLadC). We used ESM-Scan to predict conservative sequence changes in a highly conserved region of this enzyme responsible for allosteric product inhibition. Our experimental results show that the ESM-zero shot model emerges as a robust method for inferring the impact of amino acid substitutions, especially when evolutionary and functional insights are intertwined. ESM-Scan is publicly available athttps://huggingface.co/spaces/thaidaev/zsp

show abstract

SaProt: Protein Language Modeling with Structure-aware Vocabulary

Cited by 43 publications

References 52 publications

Feature Reuse and Scaling: Understanding Transfer Learning with Protein Language Models

Feature Reuse and Scaling: Understanding Transfer Learning with Protein Language Models

Pairing interacting protein sequences using masked language modeling

ESM-Scan - a tool to guide amino acid substitutions

Contact Info

Product

Resources

About