When Geometric Deep Learning Meets Pretrained Protein Language Models

Abstract

Geometric deep learning has recently achieved great success in non-Euclidean domains, and learning on 3D structures of large biomolecules is emerging as a distinct research area. However, its efficacy is largely constrained due to the limited quantity of structural data. Meanwhile, protein language models trained on substantial 1D sequences have shown burgeoning capabilities with scale in a broad range of applications. Nevertheless, no preceding studies consider combining these different protein modalities to promote the representation power of geometric neural networks. To address this gap, we make the foremost step to integrate the knowledge learned by well-trained protein language models into several state-of-the-art geometric networks. Experiments are evaluated on a variety of protein representation learning benchmarks, including protein-protein interface prediction, model quality assessment, protein-protein rigid-body docking, and binding affinity prediction, leading to an overall improvement of 20\% over baselines and the new state-of-the-art performance. Strong evidence indicates that the incorporation of protein language models' knowledge enhances geometric networks' capacity by a significant margin and can be generalized to complex tasks.

Pair-EGRET: enhancing the prediction of protein-protein interaction sites through graph attention networks and protein language models

“…Table 1: Comparison between the predictive performance of different methods in predicting pairwise PPIS of the test complexes of DBD5 and Dockground. Scores for the baseline methods on DBD5 are reported from [10,25,26]…”

“…Some recent deep learning-based methods [8,10,25,26] have demonstrated that incorporating information from both the primary amino-acid sequence and the 3D structure of a protein leads to more accurate identification of PPIS. For extracting features from the primary sequence, the recently developed protein language models [27,28], trained on large datasets of 1D amino acid sequences, have been proved to be effective [8,26,29].…”

“…Some recent deep learning-based methods [8,10,25,26] have demonstrated that incorporating information from both the primary amino-acid sequence and the 3D structure of a protein leads to more accurate identification of PPIS. For extracting features from the primary sequence, the recently developed protein language models [27,28], trained on large datasets of 1D amino acid sequences, have been proved to be effective [8,26,29]. For encoding the 3D structural information of proteins, several geometric structures have been proposed [9,30] among which Graph Neural Networks (GNN) [31] has proved to be useful for a number of methods [8,10,25].…”

A joint embedding of protein sequence and structure enables robust variant effect predictions

“…Examples of the types of data used as input include the wild-type amino acid sequence ( Lin et al, 2022; Brandes et al, 2022 ), a multiple sequence alignment (MSA) ( Ng and Henikoff, 2001; Balakrishnan et al, 2011; Lui and Tiana, 2013; Nielsen et al, 2017; Hopf et al, 2017; Riesselman et al, 2018; Laine et al, 2019 ) or the protein structure ( Boomsma and Frellsen, 2017; Jing et al, 2021a; Hsu et al, 2022 ). Some methods have combined predictions from multiple protein data types at an aggregate level ( Strokach et al, 2021; Høie et al, 2022; Cagiada et al, 2023; Nguyen and Hy, 2023 ), although some results suggest that a richer representation might be learned by combining multiple data types at the input level ( Mansoor et al, 2021; Wu et al, 2023; Wang et al, 2022; Yang et al, 2022; Chen et al, 2023; Cheng et al, 2023; Zhang et al, 2023 ).…”