“…Overall, these results suggest that the MolPROP fusion strategy is predominantly beneficial for regression tasks. Future work may explore alternative fusion strategies to improve the stability of multimodal fusion on classification tasks such as graph pretraining [ 11 , 14 ], attention mechanisms [ 33 ] or convolutional feature extraction of the language representation [ 34 ]. Fig.…”
Section: Resultsmentioning
confidence: 99%
“…This is a simple and effective strategy for exploring the fusion of language and graph representations for small molecules, however, future work may explore strategies that include hydrogens and/ or a dynamic mapping between the token and graph representations. Moreover, alternative strategies for graph and language fusion may utilize graph pretraining [11,14], an attention mechanism [33], or convolutional feature extraction of the language representation before concatenating to the graph nodes [34].…”
Section: Language and Graph Model Fusionmentioning
confidence: 99%
“…These fundamental limitations of the GNNs may explain the more variable performance on classification tasks. Future work may explore alternative graph and language fusion strategies for small molecule classification tasks that utilizes graph pretraining [11,14], an attention mechanism [33], or convolutional feature extraction of the language representation [34]. Additional strategies may include distilling the language model embeddings from ChemBERTa before concatenating to the graph nodes to improve GCN and GATv2 stability on classification tasks.…”
“…Overall, these results suggest that the MolPROP fusion strategy is predominantly beneficial for regression tasks. Future work may explore alternative fusion strategies to improve the stability of multimodal fusion on classification tasks such as graph pretraining [11,14], attention mechanisms [33] or convolutional feature extraction of the language representation [34]. Finally, we explore the learned latent embeddings representations of the MolPROP GATv2-ChemBERTa-2-77 M-MLM model by projecting the latent embeddings onto a 2D space using UMAP [44] (Fig.…”
Section: Table 3 Molprop and Baseline Model Performance On Classifica...mentioning
Pretrained deep learning models self-supervised on large datasets of language, image, and graph representations are often fine-tuned on downstream tasks and have demonstrated remarkable adaptability in a variety of applications including chatbots, autonomous driving, and protein folding. Additional research aims to improve performance on downstream tasks by fusing high dimensional data representations across multiple modalities. In this work, we explore a novel fusion of a pretrained language model, ChemBERTa-2, with graph neural networks for the task of molecular property prediction. We benchmark the MolPROP suite of models on seven scaffold split MoleculeNet datasets and compare with state-of-the-art architectures. We find that (1) multimodal property prediction for small molecules can match or significantly outperform modern architectures on hydration free energy (FreeSolv), experimental water solubility (ESOL), lipophilicity (Lipo), and clinical toxicity tasks (ClinTox), (2) the MolPROP multimodal fusion is predominantly beneficial on regression tasks, (3) the ChemBERTa-2 masked language model pretraining task (MLM) outperformed multitask regression pretraining task (MTR) when fused with graph neural networks for multimodal property prediction, and (4) despite improvements from multimodal fusion on regression tasks MolPROP significantly underperforms on some classification tasks. MolPROP has been made available at https://github.com/merck/MolPROP.
Scientific contribution
This work explores a novel multimodal fusion of learned language and graph representations of small molecules for the supervised task of molecular property prediction. The MolPROP suite of models demonstrates that language and graph fusion can significantly outperform modern architectures on several regression prediction tasks and also provides the opportunity to explore alternative fusion strategies on classification tasks for multimodal molecular property prediction.
“…Overall, these results suggest that the MolPROP fusion strategy is predominantly beneficial for regression tasks. Future work may explore alternative fusion strategies to improve the stability of multimodal fusion on classification tasks such as graph pretraining [ 11 , 14 ], attention mechanisms [ 33 ] or convolutional feature extraction of the language representation [ 34 ]. Fig.…”
Section: Resultsmentioning
confidence: 99%
“…This is a simple and effective strategy for exploring the fusion of language and graph representations for small molecules, however, future work may explore strategies that include hydrogens and/ or a dynamic mapping between the token and graph representations. Moreover, alternative strategies for graph and language fusion may utilize graph pretraining [11,14], an attention mechanism [33], or convolutional feature extraction of the language representation before concatenating to the graph nodes [34].…”
Section: Language and Graph Model Fusionmentioning
confidence: 99%
“…These fundamental limitations of the GNNs may explain the more variable performance on classification tasks. Future work may explore alternative graph and language fusion strategies for small molecule classification tasks that utilizes graph pretraining [11,14], an attention mechanism [33], or convolutional feature extraction of the language representation [34]. Additional strategies may include distilling the language model embeddings from ChemBERTa before concatenating to the graph nodes to improve GCN and GATv2 stability on classification tasks.…”
“…Overall, these results suggest that the MolPROP fusion strategy is predominantly beneficial for regression tasks. Future work may explore alternative fusion strategies to improve the stability of multimodal fusion on classification tasks such as graph pretraining [11,14], attention mechanisms [33] or convolutional feature extraction of the language representation [34]. Finally, we explore the learned latent embeddings representations of the MolPROP GATv2-ChemBERTa-2-77 M-MLM model by projecting the latent embeddings onto a 2D space using UMAP [44] (Fig.…”
Section: Table 3 Molprop and Baseline Model Performance On Classifica...mentioning
Pretrained deep learning models self-supervised on large datasets of language, image, and graph representations are often fine-tuned on downstream tasks and have demonstrated remarkable adaptability in a variety of applications including chatbots, autonomous driving, and protein folding. Additional research aims to improve performance on downstream tasks by fusing high dimensional data representations across multiple modalities. In this work, we explore a novel fusion of a pretrained language model, ChemBERTa-2, with graph neural networks for the task of molecular property prediction. We benchmark the MolPROP suite of models on seven scaffold split MoleculeNet datasets and compare with state-of-the-art architectures. We find that (1) multimodal property prediction for small molecules can match or significantly outperform modern architectures on hydration free energy (FreeSolv), experimental water solubility (ESOL), lipophilicity (Lipo), and clinical toxicity tasks (ClinTox), (2) the MolPROP multimodal fusion is predominantly beneficial on regression tasks, (3) the ChemBERTa-2 masked language model pretraining task (MLM) outperformed multitask regression pretraining task (MTR) when fused with graph neural networks for multimodal property prediction, and (4) despite improvements from multimodal fusion on regression tasks MolPROP significantly underperforms on some classification tasks. MolPROP has been made available at https://github.com/merck/MolPROP.
Scientific contribution
This work explores a novel multimodal fusion of learned language and graph representations of small molecules for the supervised task of molecular property prediction. The MolPROP suite of models demonstrates that language and graph fusion can significantly outperform modern architectures on several regression prediction tasks and also provides the opportunity to explore alternative fusion strategies on classification tasks for multimodal molecular property prediction.
“…This approach enhances the model's ability to comprehend spatial relationships within molecular structures.However, its geometric focus may not be suitable for all types of molecular attributes, especially those unrelated to spatial configurations. Lastly, MMSG (Wu et al, 2023) stands out as a multimodal learning framework for molecular graph representation. By integrating information from different modalities, such as SMILES and molecular graphs, it aims to produce more robust and versatile representations.…”
Section: Semi-supervised Learning On Graphmentioning
In recent years, graph neural network technology has revolutionized molecular graph matching methods, offering new opportunities for drug discovery. Central to this research are two key steps: latent representation learning of molecules and molecular graph matching. However, the challenge lies in effectively capturing and expressing the intricate features of molecular graphs, given their strict chemical rules and high structural complexity. To address this, we propose a gaussian-enhanced graph matching method. This approach combines a dual-channel message-passing neural network, based in the GIN algorithm, to encode both nodes and edges of molecules, enabling precise representations of molecular graphs. Additionally, an end-to-end graph similarity calculation model is introduced, assessing similarity at both the node and global levels. By integrating these evaluations, we obtain a comprehensive similarity score for molecular graph pairs. This method enhances the accuracy and interpretability of molecular structural similarity. Experimental results demonstrate the superior performance of our model compared to state-of-the-art baseline methods, marking a significant step forward in molecular property prediction.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
