Bilingual Language Model for Protein Sequence and Structure

Abstract

Advanced Artificial Intelligence (AI) enabled large language models (LLMs) to revolutionize Natural Language Processing (NLP). Their adaptation to protein sequences spawned the development of powerful protein language models (pLMs). Concurrently, AlphaFold2 broke through in protein structure prediction. For the first time, we can now systematically and comprehensively explore the dual nature of proteins that act and exist as three-dimensional (3D) machines and evolve in linear strings of one-dimensional (1D) sequences. Here, we leverage pLMs to simultaneously model both modalities by combining 1D sequences with 3D structure in one generic model. For this, we encode protein structures as token sequences using the 3Di-alphabet introduced by Foldseek. The resulting 'structure-sequence' representation is processed by a pLM to extract features and patterns. Toward this end, we constructed a non-redundant dataset from AlphaFoldDB and fine-tuned an existing pLM (ProtT5) to translate between 3Di and amino acid sequences. As a proof-of-concept for our novel approach, dubbed Protein structure-sequence T5 (ProstT5), we showed improved performance for subsequent prediction tasks, and for 'inverse folding', namely the generation of novel protein sequences adopting a given structural scaffold ('fold'). Our work showcased the potential of pLMs to tap into the information-rich protein structure revolution fueled by AlphaFold2. It paves the way for the development of tools optimizing the integration of this vast 3D structure data resource, opening new research avenues in the post AlphaFold2 era. We released our model at https://github.com/mheinzinger/ProstT5 .

ProSST: Protein Language Modeling with Quantized Structure and Disentangled Attention

“…Moreover, the recent advancements in protein folding models [2,8,29] have provided structure-based models with access to extensive datasets of protein structures. This has led to a growing interest in developing pre-training models that leverage protein structure information [7,5,27].…”

“…Similarly, the recent ProtSSN [32] model leverages ESM-2 [8] embeddings as input for the EGNN [33] model, resulting in notable advancements. Both ESM-IF1 [34] and MIF-ST [35] target inverse folding, utilizing the structure to predict corresponding protein residues, whereas ProstT5 [7] focuses on the transformation between residue sequences and their structure token sequences [6] as a pre-training objective. SaProt [5] constructs a structure-aware vocabulary using structure tokens generated by foldseek [6].…”

“…This approach not only decreases the risk of overfitting but also facilitates the incorporation of structure data into Transformer architectures. Such models include ProtsT5 [7] and SaProt [5], which build upon existing sequence models. For instance, SaProt extends the ESM model [8], while ProtsT5 evolves from the ProtT5 model [9].…”

Fine-tuning protein language models boosts predictions across diverse tasks

“…More complex architectures have also been explored 33 . Continued unsupervised training can focus models on specific protein families 34 or enrich embeddings with structural information essentially creating a bi-lingual pLM 35 . Training specialist models from scratch on smaller, specific proteins, e.g., antibodies 25,36 , seems an alternative to "continued training" (train pLM on large generic data and refine on specific proteins).…”

Domainator, a flexible software suite for domain-based annotation and neighborhood analysis, identifies proteins involved in antiviral systems

“…But on the user side, the computational demands of de novo structure prediction for query sequences can still limit the number of searches possible. Methods for converting amino acid sequences directly to Foldseek-compatible encodings, without full atomic coordinate prediction, can greatly expand the search space and scale accessible to the users 21,22 .…”