Neural network extrapolation to distant regions of the protein fitness landscape

Abstract

Machine learning (ML) has transformed protein engineering by constructing models of the underlying sequence-function landscape to accelerate the discovery of new biomolecules. ML-guided protein design requires models, trained on local sequence-function information, to accurately predict distant fitness peaks. In this work, we evaluate neural networks’ capacity to extrapolate beyond their training data. We perform model-guided design using a panel of neural network architectures trained on protein G (GB1)-Immunoglobulin G (IgG) binding data and experimentally test thousands of GB1 designs to systematically evaluate the models’ extrapolation. We find each model architecture infers markedly different landscapes from the same data, which give rise to unique design preferences. We find simpler models excel in local extrapolation to design high fitness proteins, while more sophisticated convolutional models can venture deep into sequence space to design proteins that fold but are no longer functional. Our findings highlight how each architecture’s inductive biases prime them to learn different aspects of the protein fitness landscape.

Decoding Stability and Epistasis in Human Myoglobin by Deep Mutational Scanning and Codon-level Machine Learning

“…We began with fitness scores of double mutants (considered at a codon level) as the target parameter to predict. Direct fitness prediction is a common task in protein engineering by ML (13,(32)(33)(34), however, in our input data we observed no strong correlation between double mutant fitness (F1,2) and respective single mutant fitness values (F1 & F2) or their sum (F1+F2) (Figure S8), indicating significant levels of epistasis. The task of direct F1,2 fitness prediction for variants with mutation sites spanning the entire myoglobin cDNA sequence was apparently too complex given our sparse double mutant training data, and models trained for direct F1,2 prediction performed poorly.…”

“…Prior work in the field has combined DMS and next-generation sequencing (NGS) and trained models to predict fitness of unseen in silico sequences (13,14). For example, NGS and DMS were combined to study epistasis using GB1 variant libraries containing single and double mutations (15).…”

Leveraging ancestral sequence reconstruction for protein representation learning

“…Over 25 generations of in silico evolution, LASE-based evolution converges at the predicted fitness peak within 2 minutes, whereas the equivalent ESM-1b-based evolution converges at an equivalent predicted fitness peak after 2 hours on identical hardware (Fig 3b). Recent work on protein G and immunoglobulin G has indicated that neural machines can accurately extrapolate beyond the sequence space of the training data 63 , suggesting that this approach in silico evolution may sample the PTE fitness peaks not yet discovered by directed evolution.…”

Opportunities and Challenges for Machine Learning-Assisted Enzyme Engineering

“…208,233 It is still an open question whether supervised models can extrapolate beyond their training data to predict novel proteins. 234,235 More expressive deep learning methods, such as deep kernels, 236,237 could be explored as an alternative to Gaussian processes for uncertainty quantification in BO. Overall, there is significant potential to improve ML-based protein fitness prediction to help guide the search toward proteins with ideal fitness.…”

“…At the same time, new classes of ML models should be developed for protein fitness prediction to take advantage of uncertainty and introduce helpful inductive biases for the domain. , There exist methods that take advantage of inductive biases and prior information about proteins, such as the assumption that most mutation effects are additive or incorporation of biophysical knowledge into models as priors. − Another method biases the search toward variants with fewer mutations, which are more likely to be stable and functional . Domain-specific self-supervision has been explored by training models on codons rather than amino acid sequences. ,, There are also efforts to utilize calibrated uncertainty about predicted fitnesses of proteins that lie out of the domain of previously screened proteins from the training set, but there is a need to expand and further test these methods in real settings. , It is still an open question whether supervised models can extrapolate beyond their training data to predict novel proteins. , More expressive deep learning methods, such as deep kernels, , could be explored as an alternative to Gaussian processes for uncertainty quantification in BO. Overall, there is significant potential to improve ML-based protein fitness prediction to help guide the search toward proteins with ideal fitness.…”