ProtGPT2 is a deep unsupervised language model for protein design

Abstract

Protein design aims to build novel proteins customized for specific purposes, thereby holding the potential to tackle many environmental and biomedical problems. Recent progress in Transformer-based architectures has enabled the implementation of language models capable of generating text with human-like capabilities. Here, motivated by this success, we describe ProtGPT2, a language model trained on the protein space that generates de novo protein sequences following the principles of natural ones. The generated proteins display natural amino acid propensities, while disorder predictions indicate that 88% of ProtGPT2-generated proteins are globular, in line with natural sequences. Sensitive sequence searches in protein databases show that ProtGPT2 sequences are distantly related to natural ones, and similarity networks further demonstrate that ProtGPT2 is sampling unexplored regions of protein space. AlphaFold prediction of ProtGPT2-sequences yields well-folded non-idealized structures with embodiments and large loops and reveals topologies not captured in current structure databases. ProtGPT2 generates sequences in a matter of seconds and is freely available.

From sequence to function through structure: deep learning for protein design

“…DARK3 [18] is a decoder-only model with 100M parameters trained on synthetic sequences. Following the principles of DARK3, ProtGPT2 leveraged a GPT2-like model [85] an trained on the Uniref50 dataset [83], leading to a model able to generate proteins in unexplored regions of the natural protein space, while presenting natural-like properties [16]. RITA [86] included a study on the scalability of generative Transformer models with several model-specific (e.g.…”

“…A step towards this goal would be DL models that prompted with a set of biological mechanism and industry-relevant properties (e.g., desired thermostability, aggregate viscosity, sequence length, subcellular localization, catalytic capabilities, or binding partners) output a sequence or structure satisfying the selected criteria with high precision in a timely fashion. Whilst this may not yet be possible, we present an offline attempt (i.e., not an end-to-end solution) by combining a generative DL model producing sequences [16] with an oracle discriminator DL model helping to query generated sequences for desired properties [22].…”

“…Motivated by the enormous success of structure prediction methods [8]–[11] and the recent availability of large putative protein structure databases [11], [12], some works are exploiting a structure-first approach to designing new folds and proteins [13], [14]. In contrast, others operate sequence-first , by training large generative language models on vast sequence databases [15], [16, p. 2], [17, p. 2], [18]. Advancing the field by exploiting different modalities (sequence, structure and even function) is fundamental, as no one modality may be able to explain all cell phenomena necessary for the design of biologics [19].…”

“…We offer a novel pipeline capable of generating protein sequences with tailored properties. In short, we couple ProtGPT2 [16], which generates de novo sequences, with ProtT5 [22, p. 5], which predicts properties from them in order to discriminate sequences by desired functions.…”

Predicting enzyme substrate chemical structure with protein language models

“…Recent advances in protein representations and machine learning have the potential to help accelerate orphan enzyme annotation beyond EC Class predictions. Particularly exciting are protein language models, a family of machine learning techniques adapted from the field of natural language processing (NLP) and tailored for protein sequence analysis [19][20][21] . These models take amino-acid sequences as inputs and output high-dimensional vector representations or embeddings.…”