Unified 2D and 3D Pre-Training of Molecular Representations

Zhu, Jinhua; Xia, Yingce; Wu, Lijun; Xie, Shufang; Qin, Tao; Zhou, Wengang; Li, Houqiang; Liu, Tie-Yan

doi:10.1145/3534678.3539368

Cited by 23 publications

(35 citation statements)

References 27 publications

(55 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To capture the rich information in molecular graph motifs, GROVER (Rong et al, 2020a) and MGSSL (Zhang et al, 2021) propose to predict or generate the motifs. Considering that 3D geometric information plays a vital role in predicting molecular properties, several recent works Stärk et al, 2021;Fang et al, 2022a;Zhu et al, 2022) pre-train the GNN encoders on molecular datasets with 3D geometric information. We recommend readers refer to a recent survey (Xia et al, 2022f) for more relevant literature.…”

Section: Pre-training On Moleculesmentioning

confidence: 99%

Mole-BERT: Rethinking Pre-training Graph Neural Networks for Molecules

Xia

Zhao

Hu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Recent years have witnessed the prosperity of pre-training graph neural networks (GNNs) for molecules. Typically, atom types as node attributes are randomly masked and GNNs are then trained to predict masked types as in AttrMask \citep{hu2020strategies}, following the Masked Language Modeling (MLM) task of BERT~\citep{devlin2019bert}. However, unlike MLM where the vocabulary is large, the AttrMask pre-training does not learn informative molecular representations due to small and unbalanced atom `vocabulary'. To amend this problem, we propose a variant of VQ-VAE~\citep{van2017neural} as a context-aware tokenizer to encode atom attributes into chemically meaningful discrete codes. This can enlarge the atom vocabulary size and mitigate the quantitative divergence between dominant (e.g., carbons) and rare atoms (e.g., phosphorus). With the enlarged atom `vocabulary', we propose a novel node-level pre-training task, dubbed Masked Atoms Modeling (MAM), to mask some discrete codes randomly and then pre-train GNNs to predict them. MAM also mitigates another issue of AttrMask, namely the negative transfer. It can be easily combined with various pre-training tasks to improve their performance. Furthermore, we propose triplet masked contrastive learning (TMCL) for graph-level pre-training to model the heterogeneous semantic similarity between molecules for effective molecule retrieval. MAM and TMCL constitute a novel pre-training framework, Mole-BERT, which can match or outperform state-of-the-art methods in a fully data-driven manner. We release the code at \textcolor{magenta}{\url{https://github.com/junxia97/Mole-BERT}}.

show abstract

Section: Pre-training On Moleculesmentioning

confidence: 99%

Mole-BERT: Rethinking Pre-training Graph Neural Networks for Molecules

Xia

Zhao

Hu³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…For example, Liu et al [27] and Stärk et al [27] used mutual information between 2D and 3D views for molecular pretraining, therefore the GNN is still able to produce implicit 3D information that can used to inform property predictions. Zhu et al [59] proposed a unified 2D and 3D pre-training which jointly leverages the 2D graph structure and the 3D geometry of the molecule. GEM [11] proposes a bond-angle graph and self-supervised tasks which use large-scale unlabelled molecules with coarse 3D spatial structures which can be calculated by cheminformatics tools such as RDKit.…”

Section: Pre-training On Molecular Graphmentioning

confidence: 99%

“…As shown in Figure 3, 3D PGT first splits the 3D conformer optimization into three generative tasks: bond length, bond angle and dihedral angel. By reconstructing these local descriptors that can fully describe the 3D conformer, the encoder could implicitly generate and encode 3D information in its latent vectors, which can better reflect certain molecular properties [59]. Considering the weight distribution problem of these three pre-training tasks, we design a pre-training surrogate metric to dynamically adjust the weights of each generative task, and conduct the pre-training of the model as a bi-level optimization problem: where 𝜆 𝑖 is the loss weight for pre-training task L 𝑖 .…”

Section: Automated Fusion Framework For Multiple 3d Pre-training Tasksmentioning

confidence: 99%

Automated 3D Pre-Training for Molecular Property Prediction

Wang¹,

Zhao²,

Tu³

et al. 2023

Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

View full text Add to dashboard Cite

Molecular property prediction is an important problem in drug discovery and materials science. As geometric structures have been demonstrated necessary for molecular property prediction, 3D information has been combined with various graph learning methods to boost prediction performance. However, obtaining the geometric structure of molecules is not feasible in many real-world applications due to the high computational cost. In this work, we propose a novel 3D pre-training framework (dubbed 3D PGT), which pretrains a model on 3D molecular graphs, and then fine-tunes it on molecular graphs without 3D structures. Based on fact that bond length, bond angle, and dihedral angle are three basic geometric descriptors corresponding to a complete molecular 3D conformer, we first develop a multi-task generative pre-train framework based on these three attributes. Next, to automatically fuse these three generative tasks, we design a surrogate metric using the total energy to search for weight distribution of the three pretext tasks since total energy corresponding to the quality of 3D conformer. Extensive experiments on 2D molecular graphs are conducted to demonstrate the accuracy, efficiency and generalization ability of the proposed 3D PGT compared to various pre-training baselines. CCS CONCEPTS• Computing methodologies → Neural networks; • Applied computing → Molecular structural biology.

show abstract

“…Traditional approaches based on molecular dynamics or Markov chain Monte Carlo are often computationally expensive, especially for large molecules [32]. The 3D-geometry-enhanced MPMs [132,134] show notable superiority in the downstream task of conformation generation because they can capture the entailed relations between 2D molecular graph and 3D conformation. Representative datasets for the evaluation of molecule generation include ZINC [42], ChEMBL [2] and QM9 [76].…”

Section: Molecular Generation (Mg)mentioning

confidence: 99%

A Systematic Survey of Molecular Pre-trained Models

Xia¹,

Zhu²,

Du³

et al. 2022

Preprint

View full text Add to dashboard Cite

Obtaining effective molecular representations is at the core of a series of important chemical tasks ranging from property prediction to drug design. So far, deep learning has achieved remarkable success in learning representations for molecules through automated feature learning in a data-driven fashion. However, training deep neural networks from scratch often requires sufficient labeled molecules which are expensive to acquire in real-world scenarios. To alleviate this issue, inspired by the success of the pretrain-then-finetune paradigm in natural language processing, tremendous efforts have been devoted to Molecular Pre-trained Models (MPMs), where neural networks are pre-trained using large-scale unlabeled molecular databases and then fine-tuned for diverse downstream tasks. Despite the prosperity, this field is fast-growing and a systematic roadmap is urgently needed for both methodology advancements and practical applications in both machine learning and scientific communities. To this end, this paper provides a systematic survey of pre-trained models for molecular representations. Firstly, to motivate MPMs studies, we highlight the limitations of training deep neural networks for molecular representations. Next, we systematically review recent advances on this topic from several key perspectives including molecular descriptors, encoder architectures, pre-training strategies, and applications. Finally, we identify several challenges and discuss promising future research directions.

show abstract

Unified 2D and 3D Pre-Training of Molecular Representations

Cited by 23 publications

References 27 publications

Mole-BERT: Rethinking Pre-training Graph Neural Networks for Molecules

Mole-BERT: Rethinking Pre-training Graph Neural Networks for Molecules

Automated 3D Pre-Training for Molecular Property Prediction

A Systematic Survey of Molecular Pre-trained Models

Contact Info

Product

Resources

About