Protein Design and Variant Prediction Using Autoregressive Generative Models

Abstract

A major biomedical challenge is the interpretation of genetic variation and the ability to design functional novel sequences. Since the space of all possible genetic variation is enormous, there is a concerted effort to develop reliable methods that can capture genotype to phenotype maps. State-of-art computational methods rely on models that leverage evolutionary information and capture complex interactions between residues. However, current methods are not suitable for a large number of important applications because they depend on robust protein or RNA alignments. Such applications include genetic variants with insertions and deletions, disordered proteins, and functional antibodies. Ideally, we need models that do not rely on assumptions made by multiple sequence alignments. Here we borrow from recent advances in natural language processing and speech synthesis to develop a generative deep neural network-powered autoregressive model for biological sequences that captures functional constraints without relying on an explicit alignment structure. Application to unseen experimental measurements of 42 deep mutational scans predicts the effect of insertions and deletions while matching state-of-art missense mutation prediction accuracies. We then test the model on single domain antibodies, or nanobodies, a complex target for alignment-based models due to the highly variable complementarity determining regions. We fit the model to a naïve llama immune repertoire and generate a diverse, optimized library of 10 5 nanobody sequences for experimental validation. Our results demonstrate the power of the 'alignment-free' autoregressive model in mutation effect prediction and design of traditionally challenging sequence families.

ProGen: Language Modeling for Protein Generation

“…The aforementioned work, along with O' Connell et al (2018), Boomsma & Frellsen (2017), and Greener et al (2018), all utilize explicit structural information for generative modeling, thereby are unable to fully capture the number and diversity of sequenceonly data available. Meanwhile sequence-only generative modeling have been attempted recently through residual causal dilated convolutional neural networks (Riesselman et al, 2019) and variational autoencoders (Costello & Martin, 2019). Unlike these prior works, our work on generative modeling focuses on a high-capacity language models that scale well with sequence data and can be used for controllable generation.…”

Evolutionary context-integrated deep sequence modeling for protein engineering

“…Next, we compared ECNet to other sequence modeling approaches for mutation effects prediction on a larger set of DMS datasets previously curated in [25] . We first compared to three unsupervised methods, including EVmutation [24] , DeepSequence [25] , and Autoregressive [48] . These methods trained generative models on homologous sequences and predicted the mutation effects by calculating the log-ratio of sequence probabilities of mutant and wild type sequences.…”

“…Generative models of protein sequences such as EVmutation and DeepSequence are dependent on the alignment of homologous sequences, which may introduce artifacts and lose important information caused by indels in the alignment. A generative autoregressive model was proposed by Riesselman et al [48] to predict the mutation effects in protein sequence, without the requirement of multiple sequence alignment.…”

“…We compared our method to EVmutation, DeepSequence and the Autoregressive model on the DeepSequence dataset that these methods have been tested on. The predictions made by these unsupervised approaches were collected from [25,48] . For our method, we performed five-fold cross-validation on this dataset and report the average performance over all the five folds.…”

Designing Feature-Controlled Humanoid Antibody Discovery Libraries Using Generative Adversarial Networks

“…Deep learning offers one route to better capture the complex relationships between sequence and protein behavior and has been the focus of many recent publications. [38][39][40][41][42] Within the context of discovery and libraries, the generative models such as Generative Adversarial Networks (GANs) 43,44 and autoencoder networks (AEs) 45 are of particular interest as they have been shown to be viable for generating unique sequences of proteins 46,47 and nanobodies 48 and antibody CDRs 49 . But these efforts focus on short sequences of proteins or portions of antibodies.…”