Protein language model embeddings for fast, accurate, alignment-free protein structure prediction

Abstract

All state-of-the-art (SOTA) protein structure predictions rely on evolutionary information captured in multiple sequence alignments (MSAs), primarily on evolutionary couplings (co-evolution). Such information is not available for all proteins and is computationally expensive to generate. Prediction models based on Artificial Intelligence (AI) using only single sequences as input are easier and cheaper but perform so poorly that speed becomes irrelevant. Here, we described the first competitive AI solution exclusively inputting embeddings extracted from pre-trained protein Language Models (pLMs), namely from the transformer pLM ProtT5, from single sequences into a relatively shallow (few free parameters) convolutional neural network (CNN) trained on inter-residue distances, i.e. protein structure in 2D. The major advance originated from processing the attention heads learned by ProtT5. Although these models required at no point any MSA, they matched the performance of methods relying on co-evolution. Although not reaching the very top, our lean approach came close at substantially lower costs thereby speeding up development and each future prediction. By generating protein-specific rather than family-averaged predictions, these new solutions could distinguish between structural features differentiating members of the same family of proteins with similar structure predicted alike by all other top methods.

Embeddings from protein language models predict conservation and variant effects

“…To repeat the previous speculation: embeddings might capture a reality that constrains what can be observed in evolution, and this reality is exactly what is used for the part of the SAV effect prediction that succeeds. If so, we would argue that our simpli ed method did not succeed because it predicted conservation without using MSAs, but that it captured positions biophysically "marked by constraints", i.e., residues with higher contact density in protein 3D structures (Weißenow et al 2021). This assumption would explain how predicted conservation (ProtT5cons) not using evolutionary information could predict SAV effects better than a slightly more correct approach (ConSeq) using MSAs to extract evolutionary information (Fig.…”

“…In fact, one pLM used here, namely ProtT5, has recently been shown to explicitly capture aspects of longrange inter-residue distances directly during pre-training. I.e., without ever being trained on any labeled data pLMs pick up structural constraints that allow protein 3D structure prediction from single protein sequences (Weißenow et al 2021). Another explanation for how ProtT5 embeddings capture conservation might be that pLMs picked up signals from short, frequently re-occurring sequence/structure motifs such as localization signals or catalytic sites that are more conserved than other parts of the sequence.…”

Embeddings from protein language models predict conservation and variant effects

“…In fact, the pLMs used here, namely ProtT5, has recently been shown to explicitly capture aspects of longrange inter-residue distances directly during pre-training, i.e., without ever being trained on any labeled data, pLMs pick up structural constraints that allow protein 3D structure prediction from single protein sequences (Weißenow et al 2021). Another explanation for how ProtT5 embeddings capture conservation might be that pLMs picked up signals from short, frequently re-occurring sequence/structure motifs such as localization signals or catalytic sites that are more conserved than other parts of the sequence.…”

“…To repeat the previous speculation: embeddings might capture a reality that constraints what can be observed in evolution, and this reality is exactly what is used for the part of the SAV effect prediction that succeeds. If so, we would argue that our simpli ed method did not succeed because it predicted conservation without using MSAs, but that it captured positions biophysically "marked by constraints", i.e., residues with higher contact density in protein 3D structures (Weißenow et al 2021). This assumption would explain how predicted conservation (ProtT5cons) not using evolutionary information could predict SAV effects better than a slightly more correct approach (ConSeq) using MSAs to extract evolutionary information (Fig.…”

Protein embeddings and deep learning predict binding residues for various ligand classes

“…AlphaFold 2 heavily relies on information from multiple sequence alignments (MSAs). Recent structure predictions without MSAs remain less accurate 16 . Either way, it remains unclear to which extent structure predictions could improve the prediction of binding residues beyond the unique opportunity to step up from binding residues to binding sites.…”