A Systematic Survey of Molecular Pre-trained Models

Abstract

Obtaining effective molecular representations is at the core of a series of important chemical tasks ranging from property prediction to drug design. So far, deep learning has achieved remarkable success in learning representations for molecules through automated feature learning in a data-driven fashion. However, training deep neural networks from scratch often requires sufficient labeled molecules which are expensive to acquire in real-world scenarios. To alleviate this issue, inspired by the success of the pretrain-then-finetune paradigm in natural language processing, tremendous efforts have been devoted to Molecular Pre-trained Models (MPMs), where neural networks are pre-trained using large-scale unlabeled molecular databases and then fine-tuned for diverse downstream tasks. Despite the prosperity, this field is fast-growing and a systematic roadmap is urgently needed for both methodology advancements and practical applications in both machine learning and scientific communities. To this end, this paper provides a systematic survey of pre-trained models for molecular representations. Firstly, to motivate MPMs studies, we highlight the limitations of training deep neural networks for molecular representations. Next, we systematically review recent advances on this topic from several key perspectives including molecular descriptors, encoder architectures, pre-training strategies, and applications. Finally, we identify several challenges and discuss promising future research directions.

Mole-BERT: Rethinking Pre-training Graph Neural Networks for Molecules

“…Considering that 3D geometric information plays a vital role in predicting molecular properties, several recent works Stärk et al, 2021;Fang et al, 2022a;Zhu et al, 2022) pre-train the GNN encoders on molecular datasets with 3D geometric information. We recommend readers refer to a recent survey (Xia et al, 2022f) for more relevant literature. Many above-mentioned works adopt AttrMask (Hu et al, 2020) as a fundamental pre-training sub-task.…”

Hard Sample Aware Network for Contrastive Deep Graph Clustering

“…Deep learning has been successful in many domain including computer vision (Zhou et al 2020;Wang et al 2022aWang et al , 2023, time series analysis (Xie et al 2022;Meng Liu 2021Liu et al 2022a), bioinformatics (Xia et al 2022b;Gao et al 2022;, and graph data mining (Wang et al 2020(Wang et al , 2021bZeng et al 2022Zeng et al , 2023Wu et al 2022;Duan et al 2022;Yang et al 2022b;Liang et al 2022b). Among these directions, deep graph clustering, which aims to encode nodes with neural networks and divide them into disjoint clusters, has attracted great attention in recent years.…”

Enhancing Molecular Property Prediction through Task-Oriented Transfer Learning: Integrating Universal Structural Insights and Domain-Specific Knowledge

“…The pretraining dataset primarily consists of unlabeled molecular data from extensive public databases like ChEMBL, PubChem, and ZINC . Popular pretraining strategies fall under self-supervised learning (SSL), including masked component modeling, − context prediction, replaced component detection, and contrastive learning. , SSL methods start with the molecular structure itself to reveal inherent patterns. Given the close link between molecular structures and physicochemical properties, these methods play a crucial role in predicting molecular properties. , They are frequently employed to establish a versatile pretraining model for various downstream tasks.…”