Language models generalize beyond natural proteins

Abstract

Learning the design patterns of proteins from sequences across evolution may have promise toward generative protein design. However it is unknown whether language models, trained on sequences of natural proteins, will be capable of more than memorization of existing protein families. Here we show that language models generalize beyond natural proteins to generate de novo proteins. We focus on two protein design tasks: fixed backbone design where the structure is specified, and unconstrained generation where the structure is sampled from the model. Remarkably although the models are trained only on sequences, we find that they are capable of designing structure. A total of 228 generated proteins are evaluated experimentally with high overall success rates (152/228 or 67%) in producing a soluble and monomeric species by size exclusion chromatography. Out of 152 experimentally successful designs, 35 have no significant sequence match to known natural proteins. Of the remaining 117, sequence identity to the nearest sequence match is at median 27%, below 20% for 6 designs, and as low as 18% for 3 designs. For fixed backbone design, the language model generates successful designs for each of eight experimentally evaluated artificially created fixed backbone targets. For unconstrained generation, sampled proteins cover diverse topologies and secondary structure compositions, and have high experimental success rate (71/129 or 55%). The designs reflect deep patterns linking sequence and structure, including motifs that occur in related natural structures, and motifs that are not observed in similar structural contexts in known protein families. The results show that language models, though only trained on sequences, learn a deep grammar that enables the design of protein structure, extending beyond natural proteins.

Structure-informed Language Models Are Protein Designers

“…Concurrently, Verkuil et al (2022) demonstrate that pLMs at scale can generalize beyond natural proteins to generate de novo proteins, and validate their hypothesis in silico and experimentally in great detail, in which pLMs are capable of even designing protein structure even though they only trained on sequences.…”

“…Recently, Lin et al (2022) show that sequential evolutionary knowledge learned by p LMs corresponds deeply to protein structures and thus materializes protein structure prediction from single sequences. In a concurrent work, Verkuil et al (2022) further demonstrate that p LMs at scale can synthesize de novo proteins on the basis of the deep grammars learned from large-scale native sequences, generalizing beyond natural proteins, at a high experimental success rate. Likewise, in the recent advances in generative AI algorithms in general, e .…”

“…Recently, Lin et al (2022) show that sequential evolutionary knowledge learned by pLMs corresponds deeply to protein structures and thus materializes protein structure prediction from single sequences. In a concurrent work, Verkuil et al (2022) further demonstrate that pLMs at scale can synthesize de novo proteins on the basis of the deep grammars learned from large-scale native sequences, generalizing beyond natural proteins, at a high experimental success rate. Likewise, in the recent advances in generative AI algorithms in general, e.g., VAE (Kingma & Welling, 2014), autoregressive models (Bahdanau et al, 2015;Vaswani et al, 2017), and diffusion probabilistic models (Sohl-Dickstein et al, 2015;Ho et al, 2020;Rombach et al, 2022), maximum likelihood estimation or its surrogates is the primary protocol for learning from massive native data by reconstruction.…”

Efficient and scalablede novoprotein design using a relaxed sequence space

“…The relaxed sequence hallucination method provides substantial efficiency advantages relative to the more commonly used MCMC methods. For example, recently described de-novo designed luciferase enzymes were produced by MCMC hallucination with 30,000 iterations 4 or large language model based designs requiring up to 170.000 iterations 9 . By contrast, our gradient-descent approach typically converged within < 100 iterations.…”

“…A DL based design method called 'deep network hallucination' 8 leverages this connection for a variety of protein design problems by iteratively updating a protein sequence until a desired property encoded in a mathematical loss function is obtained. Typically, when performing deep network hallucination, random mutations are applied in a Monte Carlo Markov Chain (MCMC) fashion 4,6,[9][10][11] . However, this method can be computationally inefficient due to the need for a large number of iterations.…”