Dark Proteins Important for Cellular Function

Schafferhans, Andrea; O’Donoghue, Seán I.; Heinzinger, Michael; Rost, Burkhard

doi:10.1002/pmic.201800227

Cited by 9 publications

(8 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On top, the resulting interpretations of the AI/ML might be less biased than experimental annotations. For instance, databases with annotations of protein function such as Swiss-Prot [85] and of protein structure such as PDB [50] are extremely biased by what today’s experimental techniques can handle [80], [86], [87].…”

Section: Discussionmentioning

confidence: 99%

ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Learning

Elnaggar

Heinzinger

Dallago

et al. 2020

Preprint

Self Cite

422

673

View full text Add to dashboard Cite

Motivation: Natural Language Processing (NLP) continues improving substantially through auto-regressive (AR) and auto-encoding (AE) Language Models (LMs). These LMs require expensive computing resources for self-supervised or un-supervised learning from huge unlabelled text corpora. The information learned is transferred through so-called embeddings to downstream prediction tasks. Computational biology and bioinformatics provide vast gold-mines of structured and sequentially ordered text data leading to extraordinarily successful protein sequence LMs that promise new frontiers for generative and predictive tasks at low inference cost. As recent NLP advances link corpus size to model size and accuracy, we addressed two questions: (1) To which extent can High-Performance Computing (HPC) up-scale protein LMs to larger databases and larger models? (2) To which extent can LMs extract features from single proteins to get closer to the performance of methods using evolutionary information? Methodology: Here, we trained two auto-regressive language models (Transformer-XL and XLNet) and two auto-encoder models (BERT and Albert) on 80 billion amino acids from 200 million protein sequences (UniRef100) and one language model (Transformer-XL) on 393 billion amino acids from 2.1 billion protein sequences taken from the Big Fat Database (BFD), today's largest set of protein sequences (corresponding to 22- and 112-times, respectively of the entire English Wikipedia). The LMs were trained on the Summit supercomputer, using 936 nodes with 6 GPUs each (in total 5616 GPUs) and one TPU Pod, using V3-512 cores. Results: We validated the feasibility of training big LMs on proteins and the advantage of up-scaling LMs to larger models supported by more data. The latter was assessed by predicting secondary structure in three- and eight-states (Q3=75-83, Q8=63-72), localization for 10 cellular compartments (Q10=74) and whether a protein is membrane-bound or water-soluble (Q2=89). Dimensionality reduction revealed that the LM-embeddings from unlabelled data (only protein sequences) captured important biophysical properties of the protein alphabet, namely the amino acids, and their well orchestrated interplay in governing the shape of proteins. In the analogy of NLP, this implied having learned some of the grammar of the language of life realized in protein sequences. The successful up-scaling of protein LMs through HPC slightly reduced the gap between models trained on evolutionary information and LMs. Additionally, our results highlighted the importance of bi-directionality when processing proteins as the uni-directional TransformerXL was outperformed by its bi-directional counterparts. Availability ProtTrans: <a href="https://github.com/agemagician/ProtTrans">https://github.com/agemagician/ProtTrans</a>

show abstract

Section: Discussionmentioning

confidence: 99%

ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Learning

Elnaggar

Heinzinger

Dallago

et al. 2020

Preprint

Self Cite

422

673

View full text Add to dashboard Cite

show abstract

“…Secondly, evolutionary information is still missing for some proteins, e.g. for proteins with substantial intrinsically disordered regions [15, 37, 38], or the entire Dark Proteome [39] full of proteins that are less-well studied but important for function [40].…”

Section: Introductionmentioning

confidence: 99%

Modeling aspects of the language of life through transfer-learning protein sequences

et al. 2019

Self Cite

View full text Add to dashboard Cite

BackgroundPredicting protein function and structure from sequence is one important challenge for computational biology. For 26 years, most state-of-the-art approaches combined machine learning and evolutionary information. However, for some applications retrieving related proteins is becoming too time-consuming. Additionally, evolutionary information is less powerful for small families, e.g. for proteins from the Dark Proteome. Both these problems are addressed by the new methodology introduced here.ResultsWe introduced a novel way to represent protein sequences as continuous vectors (embeddings) by using the language model ELMo taken from natural language processing. By modeling protein sequences, ELMo effectively captured the biophysical properties of the language of life from unlabeled big data (UniRef50). We refer to these new embeddings as SeqVec (Sequence-to-Vector) and demonstrate their effectiveness by training simple neural networks for two different tasks. At the per-residue level, secondary structure (Q3 = 79% ± 1, Q8 = 68% ± 1) and regions with intrinsic disorder (MCC = 0.59 ± 0.03) were predicted significantly better than through one-hot encoding or through Word2vec-like approaches. At the per-protein level, subcellular localization was predicted in ten classes (Q10 = 68% ± 1) and membrane-bound were distinguished from water-soluble proteins (Q2 = 87% ± 1). Although SeqVec embeddings generated the best predictions from single sequences, no solution improved over the best existing method using evolutionary information. Nevertheless, our approach improved over some popular methods using evolutionary information and for some proteins even did beat the best. Thus, they prove to condense the underlying principles of protein sequences. Overall, the important novelty is speed: where the lightning-fast HHblits needed on average about two minutes to generate the evolutionary information for a target protein, SeqVec created embeddings on average in 0.03 s. As this speed-up is independent of the size of growing sequence databases, SeqVec provides a highly scalable approach for the analysis of big data in proteomics, i.e. microbiome or metaproteome analysis.ConclusionTransfer-learning succeeded to extract information from unlabeled sequence databases relevant for various protein prediction tasks. SeqVec modeled the language of life, namely the principles underlying protein sequences better than any features suggested by textbooks and prediction methods. The exception is evolutionary information, however, that information is not available on the level of a single sequence.

show abstract

“…checked the hypothesis that the darkest members of the dark human proteome (i.e., proteins for which there is no any structural information (experimental or predicted based on the homology‐based models) for any part of the amino acid sequence) are less abundantly expressed than the non‐dark proteins (in which at least part of the sequence has a known or inferred structure) . The output of this study was completely unexpected, since the authors revealed that there is no difference in the expression levels of dark and non‐dark proteins, indicating that dark proteins are massively produced at both mRNA and protein levels . This is at stark contrast to the results of the comprehensive analysis of the modern life science literature conducted by Sinha et al.…”

mentioning

confidence: 86%

“…The authors showed that the overwhelming majority of the scientific literature is about less than 5000 “elite” genes, whereas thousands of human proteins have never been mentioned at all . Therefore, despite massive production of dark proteins, which is clearly a reflection of their biological importance, the function discovery rate remains very low for the big portion of human proteome that clearly continues to keep its functionality in darkness. To find a potential correlation between structural coverage, degree of putative intrinsic disorder, and predicted propensity for structure determination at th proteome level, Hu et al.…”

mentioning

confidence: 99%

Bringing Darkness to Light: Intrinsic Disorder as a Means to Dig into the Dark Proteome

Uversky

2018

Proteomics

View full text Add to dashboard Cite

Articles assembled to this Special Issue use the perspective of protein intrinsic disorder to highlight different aspects of the dark proteome (i.e., a part of protein universe that includes proteins, which are not amenable to experimental structure determination by existing means and inaccessible to homology modeling) and some of its components. They illustrate that IDPs/IDPRs do not only serve as important constituents of the biological dark matter, but clearly act as the dark horse of the protein universe, being almost completely unknown or at least very little known in a recent past, but suddenly emerging to prominence.About two decades ago, when I started working on α-synuclein, a seditious idea of the possible commonness of structure-less proteins with important biological functions struck my mind. It was kind of a revelation. However, to be completely honest, this revelation should have happened a few years earlier. In fact, αsynuclein represented already a third case of natively unfolded proteins I personally dealt with, with two other highly disordered but functional proteins being prothymosin α [1] and phosphodiesterase γ subunit. [2] Then, there were several personally witnessed cases of "broken" proteins, which did not have unique 3-structure as a result of the removal of natural ligands (e.g., αfetoprotein [3] or many other globular proteins [4] ) or due to some mutations (e.g., mutated [5] or circularly permuted dihydrofolate reductase [6] ). But, of course, these cases of "broken" proteins do not count, since all of them were shown to fold in the presence of natural ligands or substrates and therefore could be classified as underfolded globular proteins.

show abstract

Dark Proteins Important for Cellular Function

Cited by 9 publications

References 18 publications

ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Learning

ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Learning

Modeling aspects of the language of life through transfer-learning protein sequences

Bringing Darkness to Light: Intrinsic Disorder as a Means to Dig into the Dark Proteome

Contact Info

Product

Resources

About