One Transformer Can Understand Both 2D & 3D Molecular Data

Abstract

Unlike vision and language data which usually has a unique format, molecules can naturally be characterized using different chemical formulations. One can view a molecule as a 2D graph or define it as a collection of atoms located in a 3D space. For molecular representation learning, most previous works designed neural networks only for a particular data format, making the learned models likely to fail for other data formats. We believe a general-purpose neural network model for chemistry should be able to handle molecular tasks across data modalities. To achieve this goal, in this work, we develop a novel Transformer-based Molecular model called Transformer-M, which can take molecular data of 2D or 3D formats as input and generate meaningful semantic representations. Using the standard Transformer as the backbone architecture, Transformer-M develops two separated channels to encode 2D and 3D structural information and incorporate them with the atom features in the network modules. When the input data is in a particular format, the corresponding channel will be activated, and the other will be disabled. By training on 2D and 3D molecular data with properly designed supervised signals, Transformer-M automatically learns to leverage knowledge from different data modalities and correctly capture the representations. We conducted extensive experiments for Transformer-M. All empirical results show that Transformer-M can simultaneously achieve strong performance on 2D and 3D tasks, suggesting its broad applicability. The code and models will be made publicly available at https://github.com/lsj2408/Transformer-M.

Exploring Chemical Reaction Space with Machine Learning Models: Representation and Feature Perspective

“…Nevertheless, inspiration can be drawn from molecule pretraining frameworks that encode 3D information even when only molecular graphs are available. Broadly speaking, these methods , align embeddings or minimize property predictions between 3D models and graph-based models, the latter of which do not necessitate 3D coordinates as input. In the realm of chemical reactions, data sets incorporating simulated reactive processes have been developed .…”

Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers

“…We compare our FLT with the regular Performer without RPE. For the FLT model, we consider to approximate RPE-masks based on Gaussian basis functions, which are popularly used in neural networks for molecular modeling (Gasteiger et al, 2021;Shi et al, 2022;Luo et al, 2022a). Specifically, the RPE-mask is defined as N = [f (r i − r j )] i,j=1,...,L ∈ R L×L , where r i ∈ R 3 is the position of the i-th input atom, L is the total number of input atom, and…”

Automated 3D Pre-Training for Molecular Property Prediction

“…GEM [11] proposes a bond-angle graph and self-supervised tasks which use large-scale unlabelled molecules with coarse 3D spatial structures which can be calculated by cheminformatics tools such as RDKit. Inspired by the encoding method of Graphormer for 2D molecular graph, Transformer-M [29] further encodes 3D spatial distance into attention bias, which can take molecular data of 2D or 3D formats as input.…”

“…Baselines. Because the label of test datasets is officially hidden, we compared the results of validation with the top-tier method on the OGB leaderboard 2 , which includes GRPE [34], TokenGT [20], EGT [18] , GPS [40], GEM-2 [26], Vis-Net [52] and Transformer-M [29]. In addition, we also compared GPS++ [30], the winner solution at OGB LSC @ NeruIPS 2022 Challenge.…”