Efficient generative modeling of protein sequences using simple autoregressive models

Abstract

Generative models emerge as promising candidates for novel sequence-data driven approaches to protein design, and for the extraction of structural and functional information about proteins deeply hidden in rapidly growing sequence databases. Here we propose simple autoregressive models as highly accurate but computationally efficient generative sequence models. We show that they perform similarly to existing approaches based on Boltzmann machines or deep generative models, but at a substantially lower computational cost (by a factor between 102 and 103). Furthermore, the simple structure of our models has distinctive mathematical advantages, which translate into an improved applicability in sequence generation and evaluation. Within these models, we can easily estimate both the probability of a given sequence, and, using the model’s entropy, the size of the functional sequence space related to a specific protein family. In the example of response regulators, we find a huge number of ca. 1068 possible sequences, which nevertheless constitute only the astronomically small fraction 10−80 of all amino-acid sequences of the same length. These findings illustrate the potential and the difficulty in exploring sequence space via generative sequence models.

Epistatic models predict mutable sites in SARS-CoV-2 proteins and epitopes

“…Two reasons are possible: 1) In limited datasets of functional sequences, undersampled neutral variants may appear deleterious and 2) mutations without effect on expression may be deleterious for other phenotypes contributing to protein fitness. This observation agrees with what has been observed across other protein families ( 30 – 32 ) where phenotypes better describing fitness also correlate better to sequence-based predictions.…”

Correlations from structure and phylogeny combine constructively in the inference of protein partners from sequences

“…Pearson correlation coefficients are 0.994 and 0.995 for panels A and B, respectively, and linear fits yield intercepts of 0.01 and 0 and slopes of 0.97 and 1 for panels A and B, respectively. The two-body frequencies shown for data set 1 include a reweighting of close sequences with Hamming distances under 0.2 [6,7], since bmDCA and arDCA aim to match these reweighted frequencies [63,64]. The fraction of correctly predicted partner pairs is shown versus the number of sequence pairs AB in the training set for data generated under the minimal model, and for data generated using models inferred from this generated data, either by bmDCA [63] (panel A) or by arDCA [64] (panel B).…”

“…There are q = 21 states, namely the 20 natural amino acids and the alignment gap. We use state-of-the-art methods that have good generative properties, namely bmDCA [16,63] and arDCA [64]. In practice, we employ bmDCA with its default parameters for q = 21, and with default parameters except t wait,0 = 1000 and ∆t 0 = 100 for q = 2 (motivated by the faster equilibration observed for q = 2).…”

“…We employ generative models inferred on natural paired HK-RR sequences (see above) to generate realistic synthetic data either without and with phylogeny. While bmDCA infers a Potts model, arDCA directly infers the distribution of probability of sequences [64]. First, to generate contact-only data, we employ a Markov Chain Monte Carlo procedure for bmDCA (equilibrium is considered reached after 10 6 accepted mutations for q = 21, or 10 5 for q = 2), while we directly sample independent sequences from the inferred distribution for arDCA.…”

“…To address this question, we consider a data set composed of 23,633 interacting pairs of natural sequences of histidine kinases (HK) and response regulators (RR) from the P2CS database [61,62]. To gain further insight into the importance of phylogenetic signal, we infer generative models of this paired sequence alignment using two state-of-the-art methods, bmDCA [63] and arDCA [64] (see Methods), and we generate data from them, either without phylogeny or along a phylogeny inferred from the natural alignment (see Methods, and Ref. [49]).…”