Dual-view Molecule Pre-training

Abstract

Inspired by its success in natural language processing and computer vision, pretraining has attracted substantial attention in cheminformatics and bioinformatics, especially for molecule based tasks. A molecule can be represented by either a graph (where atoms are connected by bonds) or a SMILES sequence (where depth-first-search is applied to the molecular graph with specific rules). Existing works on molecule pre-training use either graph representations only or SMILES representations only. In this work, we propose to leverage both the representations and design a new pre-training algorithm, dual-view molecule pre-training (briefly, DMP), that can effectively combine the strengths of both types of molecule representations. The model of DMP consists of two branches: a Transformer branch that takes the SMILES sequence of a molecule as input, and a GNN branch that takes a molecular graph as input. The training of DMP contains three tasks:(1) predicting masked tokens in a SMILES sequence by the Transformer branch, (2) predicting masked atoms in a molecular graph by the GNN branch, and (3) maximizing the consistency between the two high-level representations output by the Transformer and GNN branches separately. After pre-training, we can use either the Transformer branch (this one is recommended according to empirical results), the GNN branch, or both for downstream tasks. DMP is tested on nine molecular property prediction tasks and achieves state-of-the-art performances on seven of them. Furthermore, we test DMP on three retrosynthesis tasks and achieve state-of-the-result on the USPTO-full dataset. Our code will be released soon.

Improving Molecular Contrastive Learning via Faulty Negative Mitigation and Decomposed Fragment Contrast

“…The model is pre-trained on approximately 10 million unique unlabeled molecules from following previous molecule SSL works. 39,40,49,53 Comparing to random split, scaffold split provides a more challenging yet more realistic setting to benchmark molecular property predictions. 28 During fine-tuning, the model is only trained on the train set and leverages the validation set to select the best-performing model.…”

“…49,[51][52][53] Few works have investigated motif-level CL for molecules, which learns a table of frequently-occurring motif embeddings and trains a sampler to generate informative subgraphs for CL 50. Though such a method shows performance enhancement on various benchmarks, the learned sampler may not cover all the unique substructures in the large molecule dataset.…”

Masked Molecule Modeling: A New Paradigm of Molecular Representation Learning for Chemistry Understanding

“…In this way, Chemformer is trained to reconstruct the original SMILES from a corrupted input by randomly masking a part of characters in the SMILES. DMP [8] employs a two-branch model (Transformer and GNN) and involves self-supervised learning that masks atoms on a single molecule and is trained to reconstruct them. Experimental results show that self-supervised learning is an effective way to improve the performance of retrosynthesis tasks.…”

“…Learning good deep representations for molecules has been vastly investigated in the literature. Since collecting labeled datasets is labor-intensive and costly, there is an ever-increasing trend in learning molecular representations in a self-supervised manner [3][4][5][6][7][8]73]. Inspired by the great success of Masked Auto-Encoder (MAE) on understanding of natural language [25,26] and image [81][82][83], there is a growing body of exploratory works on chemical understanding by masking characters in SMILES, or atoms and bonds on a molecular graph.…”

“…Good representations of molecules are useful in chemical understanding [1][2][3][4][5][6][7][8]. Traditionally, molecules are represented as structure diagrams with bonds and atoms [9][10][11].…”