Light Attention Predicts Protein Location from the Language of Life

Rost

2021

Preprint

Self Cite

All state-of-the-art (SOTA) protein structure predictions rely on evolutionary information captured in multiple sequence alignments (MSAs), primarily on evolutionary couplings (co-evolution). Such information is not available for all proteins and is computationally expensive to generate. Prediction models based on Artificial Intelligence (AI) using only single sequences as input are easier and cheaper but perform so poorly that speed becomes irrelevant. Here, we described the first competitive AI solution exclusively inputting embeddings extracted from pre-trained protein Language Models (pLMs), namely from the transformer pLM ProtT5, from single sequences into a relatively shallow (few free parameters) convolutional neural network (CNN) trained on inter-residue distances, i.e. protein structure in 2D. The major advance originated from processing the attention heads learned by ProtT5. Although these models required at no point any MSA, they matched the performance of methods relying on co-evolution. Although not reaching the very top, our lean approach came close at substantially lower costs thereby speeding up development and each future prediction. By generating protein-specific rather than family-averaged predictions, these new solutions could distinguish between structural features differentiating members of the same family of proteins with similar structure predicted alike by all other top methods.

Section: Introduction mentioning

confidence: 99%

Protein language model embeddings for fast, accurate, alignment-free protein structure prediction

Weißenow

Rost

2021

Preprint

Self Cite

“…On the other hand, it was already shown that methods using embeddings from protein LMs as input get very close or even outperform methods that use evolutionary constraints defined by MSAs as input (Elnaggar et al 2021;Rao et al 2020;Stärk et al 2021). Therefore, we can currently only conclude that the simplest approach of using reconstruction probabilities from ProtBert without further processing are not readily suitable to scan the mutational landscape of proteins.…”

Section: Discussionmentioning

confidence: 94%

“…1 in (Elnaggar et al 2021)). Embeddings have been used successfully as exclusive input to predicting secondary structure and subcellular localization at performance levels almost reaching (Alley et al 2019;Heinzinger et al 2019;Rives et al 2021) or even exceeding (Elnaggar et al 2021;Stärk et al 2021) state-of-the-art methods using evolutionary information from MSAs as input. Embeddings can even substitute sequence similarity for homology-based annotation transfer (Littmann et al 2021a;Littmann et al 2021b).…”

Section: Introductionmentioning

confidence: 99%

“…Figure 1 in )). Embeddings have succeeded as exclusive input to predicting secondary structure and subcellular localization at performance levels almost reaching (Alley et al 2019;Heinzinger et al 2019; or even exceeding Stärk et al 2021) state-of-the-art (SOTA) methods using EI from MSAs as input. Embeddings can even substitute sequence similarity for homology-based annotation transfer (Littmann et al 2021a;Littmann et al 2021b) and can predict the effect of mutations on protein-protein interactions (Zhou et al 2020).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Embeddings from protein language models predict conservation and variant effects

Marquet

Olenyi

et al. 2021

Preprint

Self Cite

The emergence of SARS-CoV-2 variants stressed the demand for tools allowing to interpret the effect of single amino acid variants (SAVs) on protein function. While Deep Mutational Scanning (DMS) sets continue to expand our understanding of the mutational landscape of single proteins, the results continue to challenge analyses. Protein Language Models (pLMs) use the latest deep learning (DL) algorithms to leverage growing databases of protein sequences. These methods learn to predict missing or masked amino acids from the context of entire sequence regions. Here, we used pLM representations (embeddings) to predict sequence conservation and SAV effects without multiple sequence alignments (MSAs). Embeddings alone predicted residue conservation almost as accurately from single sequences as ConSeq using MSAs (two-state Matthew Correlation Coefficient – MCC - for ProtT5 embeddings of 0.596 ± 0.006 vs. 0.608 ± 0.006 for ConSeq). Inputting the conservation prediction along with BLOSUM62 substitution scores and pLM mask reconstruction probabilities into a simplistic logistic regression (LR) ensemble for Variant Effect Scoring without alignments (VESPA) predicted SAV effect magnitude without any optimization on DMS data. Comparing predictions for a standard set of 39 DMS experiments to other methods (incl. ESM-1v, DeepSequence, and GEMME) revealed our approach as competitive with the state-of-the-art (SOTA) methods using MSA input. No method outperformed all others, neither consistently nor in a statistically significant manner, independently of the performance measure applied (incl. two-state accuracy: Q2, MCC, Spearman and Pearson correlation). Lastly, we investigated binary effect predictions on DMS experiments for four human proteins. Overall, embedding-based methods have become competitive with methods relying on MSAs for SAV effect prediction at a fraction of the costs in computing/energy. Our method predicted SAV effects for the entire human proteome (~ 20k proteins) within 40 minutes on one Nvidia Quadro RTX 8000. All methods and data sets are freely available for local and online execution through bioembeddings.com, https://github.com/Rostlab/VESPA, and PredictProtein.

“…1 in (Elnaggar et al 2021)). Embeddings have succeeded as exclusive input to predicting secondary structure and subcellular location at performance levels almost reaching (Alley et al 2019;Heinzinger et al 2019;Rives et al 2021) or even exceeding (Elnaggar et al 2021;Littmann et al 2021c;Stärk et al 2021) state-of-the-art (SOTA) methods using EI from MSAs as input. Embeddings even succeed in substituting sequence similarity for homology-based annotation transfer (Littmann et al 2021a;Littmann et al 2021b) and in predicting the effect of mutations on protein-protein interactions (Zhou et al 2020).…”

Section: Introductionmentioning

confidence: 99%

Embeddings from protein language models predict conservation and variant effects

Marquet

Olenyi

et al. 2021

Preprint

Self Cite

The emergence of SARS-CoV-2 variants stressed the demand for tools allowing to interpret the effect of single amino acid variants (SAVs) on protein function. While Deep Mutational Scanning (DMS) sets continue to expand our understanding of the mutational landscape of single proteins, the results continue to challenge analyses. Protein Language Models (pLMs) use the latest deep learning (DL) algorithms to leverage growing databases of protein sequences. These methods learn to predict missing or masked amino acids from the context of entire sequence regions. Here, we used pLM representations (embeddings) to predict sequence conservation and SAV effects without multiple sequence alignments (MSAs). Embeddings alone predicted residue conservation almost as accurately from single sequences as ConSeq using MSAs (two-state Matthews Correlation Coefficient – MCC - for ProtT5 embeddings of 0.596±0.006 vs. 0.608±0.006 for ConSeq). Inputting the conservation prediction along with BLOSUM62 substitution scores and pLM mask reconstruction probabilities into a simplistic logistic regression (LR) ensemble for Variant Effect Score Prediction without Alignments (VESPA) predicted SAV effect magnitude without any optimization on DMS data. Comparing predictions for a standard set of 39 DMS experiments to other methods (incl. ESM-1v, DeepSequence, and GEMME) revealed our approach as competitive with the state-of-the-art (SOTA) methods using MSA input. No method outperformed all others, neither consistently nor statistically significantly, independently of the performance measure applied (Spearman and Pearson correlation). Lastly, we investigated binary effect predictions on DMS experiments for four human proteins. Overall, embedding-based methods have become competitive with methods relying on MSAs for SAV effect prediction at a fraction of the costs in computing/energy. Our method predicted SAV effects for the entire human proteome (~20k proteins) within 40 minutes on one Nvidia Quadro RTX 8000. All methods and data sets are freely available for local and online execution through bioembeddings.com, https://github.com/Rostlab/VESPA, and PredictProtein.