SMICLR: Contrastive Learning on Multiple Molecular Representations for Semisupervised and Unsupervised Representation Learning

Abstract

Machine learning as a tool for chemical space exploration broadens horizons to work with known and unknown molecules. At its core lies molecular representation, an essential key to improve learning about structure–property relationships. Recently, contrastive frameworks have been showing impressive results for representation learning in diverse domains. Therefore, this paper proposes a contrastive framework that embraces multimodal molecular data. Specifically, our approach jointly trains a graph encoder and an encoder for the simplified molecular-input line-entry system (SMILES) string to perform the contrastive learning objective. Since SMILES is the basis of our method, i.e., we built the molecular graph from the SMILES, we call our framework as SMILES Contrastive Learning (SMICLR). When stacking a nonlinear regressor on the SMICLR’s pretrained encoder and fine-tuning the entire model, we reduced the prediction error by, on average, 44% and 25% for the energetic and electronic properties of the QM9 data set, respectively, over the supervised baseline. We further improved our framework’s performance when applying data augmentations in each molecular-input representation. Moreover, SMICLR demonstrated competitive representation learning results in an unsupervised setting.

Complementary multi-modality molecular self-supervised learning via non-overlapping masking for property prediction

“…1 , the rational combination of SMILES and graph holds promise for enhancing molecular representation performance. Most existing approaches often rely on the contrastive method, such as SMICLR [ 35 ], DVMP [ 21 ] and MOCO [ 24 ], which focus on the same two modalities as we do but they neglect the fine-grained interactions across different modalities. A concurrent work, UniMAP [ 36 ], is a generative pre-training based on mask reconstruction, but it only performs simple mask reconstruction without a specific design of masking strategy, so it still cannot fully leverage the complementary information interactions.…”

Multilevel contrast strategy for unpaired image-to-image translation

“…In addition, SimCLR 41 shows that adding a nonlinear transformation between the encoder and loss function enhances the encoder's capabilities by preventing the loss of valuable information for downstream tasks. Some works [34][35][36]47,48 employ pixelwise contrast loss, where they select the pixels in feature maps as samples for contrastive loss. PixPro 35 and VADeR 47 select negative keys based on their location in the image, whereas SetSim 34 and DenseCLIP 36 choose the pixels that are semantically similar with queries.…”

Enhancing Generalizability in Protein–Ligand Binding Affinity Prediction with Multimodal Contrastive Learning

“…For a biased model, it may correctly predict the binding affinity based on the incorrect model. Noteworthy, contrastive learning has showcased competitive results in tasks like small molecule property prediction, − sequences-based prediction of drug–target interactions, similarity-based virtual screening, reaction classification, and enzyme function prediction . Contrastive learning can be applied in a multimodal setting, which enhances the learning of joint representations from varied modalities and thus bolsters model’s performance. , Given those considerations, we have therefore employed contrastive learning to reduce the embedding distance of both protein–ligand representation modalities and expand the distance from incorrect poses in a shared latent space, thereby improving the protein–ligand representation.…”