Learn molecular representations from large-scale unlabeled molecules for drug discovery

Abstract

How to produce expressive molecular representations is a fundamental challenge in AIdriven drug discovery. Graph neural network (GNN) has emerged as a powerful technique for modeling molecular data. However, previous supervised approaches usually suffer from the scarcity of labeled data and have poor generalization capability. Here, we proposed a novel Molecular Pre-training Graph-based deep learning framework, named MPG, that leans molecular representations from large-scale unlabeled molecules. In MPG, we proposed a powerful MolGNet model and an effective self-supervised strategy for pre-training the model at both the node and graph-level. After pre-training on 11 million unlabeled molecules, we revealed that MolGNet can capture valuable chemistry insights to produce interpretable representation. The pre-trained MolGNet can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of drug discovery tasks, including molecular properties prediction, drug-drug interaction, and drug-target interaction, involving 13 benchmark datasets. Our work demonstrates that MPG is promising to become a novel approach in the drug discovery pipeline.

Dual-view Molecule Pre-training

“…Intuitively, molecules with the same scaffold share similar architectures, and therefore are expected to be close in the highlevel representation space. Following [32], we choose ten representative scaffolds (denoted as S) 6 and then randomly select 200k compounds. For each compound whose scaffold lies in S, we obtain its three representations: one from GNN pre-training with MLM only, one from Transformer pretraining with MLM only, and the third one from the Transformer branch of our DMP pre-training.…”

“…GNN-based pre-training: Hu et al [23] used a GNN to encode the input molecule, and proposed two pre-training strategies, where we should either recover the masked attributes of the input (e.g., atom type), or use constrastive learning [17,2] to minimize the difference between two subgraphs of within a molecule. A similar idea also exists in Li et al [32]. Wang et al [53] applied contrastive learning across different molecules and proposed MolCLR, where a molecule should be similar to an augmented version of itself while dissimilar to others.…”

“…On one hand, following [53], we use the official training, validation and test sets provided by DeepChem 5 , whose performance is relatively stable and reproducible. On the other hand, to compare with [46,32], we follow their ways to split the data and conduct another group of experiments. Specifically, we use scaffold splitter to generate the training (80%), validation (10%) and test (10%) sets with 3 different seeds.…”

“…Specifically, we use scaffold splitter to generate the training (80%), validation (10%) and test (10%) sets with 3 different seeds. We compare with [46,32] on three classification and three regression tasks.…”

“…The other two are GNN-based method, D-MPNN [57] and MGCNN [36], which are specifically designed for molecular property prediction. We also compare our model with previous pre-training methods, including Hu et al [23], MolCLR [53], GROVER [46] and MPG [32].…”