Prior knowledge facilitates low homologous protein secondary structure prediction with DSM distillation

Abstract

Motivation Protein secondary structure prediction (PSSP) is one of the fundamental and challenging problems in the field of computational biology. Accurate PSSP relies on sufficient homologous protein sequences to build the multiple sequence alignment (MSA). Unfortunately, many proteins lack homologous sequences, which results in the low quality of MSA and poor performance. In this paper, we propose the novel DSM-Distil to tackle this issue, which takes advantage of the pretrained BERT and exploits the knowledge distillation on the newly designed dynamic scoring matrix (DSM) features. Specifically, we propose the dynamic scoring matrix (DSM) to replace the widely used profile and PSSM features. DSM could automatically dig for the suitable feature for each residue, based on the original profile. Namely, DSM-Distil not only could adapt to the low homologous proteins but also is compatible with high homologous ones. Thanks to the dynamic property, DSM could adapt to the input data much better and achieve higher performance. Moreover, to compensate for low-quality MSA, we propose to generate the pseudo-DSM from a pretrained BERT model and aggregate it with the original DSM by adaptive residue-wise fusion, which helps to build richer and more complete input features. In addition, we propose to supervise the learning of low-quality DSM features by using high-quality ones. To achieve this, a novel teacher-student model is designed to distill the knowledge from proteins with high homologous sequences to that of low ones. Combining all the proposed methods, our model achieves the new state-of-the-art performance for low homologous proteins. Results Compared with the previous state-of-the-art method “Bagging”, DSM-Distil achieves an improvement about 5% and 7.3% improvement for proteins with MSA count ≤ 30 and extremely low homologous cases respectively. We also compare DSM-Distil with Alphafold2 which is a state-of-the-art framework for protein structure prediction. DSM-Distil outperforms Alphafold2 by 4.1% on extremely low-quality MSA on 8-state secondary structure prediction. Moreover, we release a large-scale up-to-date test dataset BC40 for low-quality MSA structure prediction evaluation. Availability and implementation BC40 dataset: https://drive.google.com/drive/folders/15vwRoOjAkhhwfjDk6-YoKGf4JzZXIMC HardCase dataset: https://drive.google.com/drive/folders/1BvduOr2b7cObUHy6GuEWk-aUkKJgzTUv Code: https://github.com/qinwang-ai/DSM-Distil

Transformer-based deep learning for predicting protein properties in the life sciences

“…There are Transformer models that utilize evolutionary information extracted from MSAs during the pre-training stage, but pre-training is mostly done as a one-off process, and representation for new proteins is extracted using only the pretrained hidden states of the Transformer models. MSA tools generate alignment by searching homologs from the entire UniProt database, time-consuming ( Hong et al, 2021 ) process, whereby generating embeddings using protein language models is less cumbersome but it also builds richer and more complete features for low homologous proteins ( Wang et al, 2022 ).…”

Leveraging transformers‐based language models in proteome bioinformatics

“…The generated feature list can be used for any tuning task in proteome bioinformatics. Some of the common applications of NLP models in proteomics are protein function prediction [10,11], protein-protein interaction prediction [20,21], protein structure prediction [22], drug discovery [23], etc. For instance, protein-protein interaction prediction aids in understanding cellular processes by identifying potential interactions between proteins, enabling the exploration of complex molecular networks and signaling pathways within cells.…”

“…Wang et al proposed DSM-Distil, a transformer-based model using DSM distillation to facilitate low homologous protein secondary structure prediction [22]. Furthermore, Villegas-Morcillo et al conducted a transformers model using word embeddings for fold prediction [71].…”

Deep learning for protein secondary structure prediction: Pre and post-AlphaFold

“…The experiment result showed that using PSSM-Distil instead of the standard PSSM increased the accuracy of PSSP in low-quality PSSM cases in BC40 and CB513 datasets. Subsequently, the same researchers also proposed a dynamic scoring matrix (DSM)-Distil to replace PSSM and other widely used features [165] . This feature leveraged the pre-trained BERT to construct a dynamic scoring matrix (DSM) and performed knowledge distillation on the DSM.…”