Chemformer: a pre-trained transformer for computational chemistry

Abstract

Transformer models coupled with Simplified Molecular Line Entry System (SMILES) have recently proven to be a powerful combination for solving challenges in cheminformatics. These models, however, are often developed specifically for a single application and can be very resource-intensive to train. In this work we present Chemformer model – a Transformerbased model which can be quickly applied to both sequence-to-sequence and discriminative cheminformatics tasks. Additionally, we show that self-supervised pre-training can improve performance and significantly speed up convergence on downstream tasks. On direct synthesis and retrosynthesis prediction benchmark datasets we publish state-of-the-art results for top- 1 accuracy. We also improve on existing approaches for a molecular optimisation task and show that Chemformer can optimise on multiple discriminative tasks simultaneously. Models, datasets and code will be made available after publication.

Root-aligned SMILES: a tight representation for chemical reaction prediction

“…It is clear that our method outperforms others in most cases. Although Chemformer 38 uses much more model parameters and data than ours for pretraining, our method still obtains better results with the exception of top-1 accuracy. In different settings, the top-5 accuracy of our method is equal to or even higher than the top-20 accuracy of MEGAN, which fully illustrates the high efficiency of our method.…”

RetroComposer: Composing Templates for Template-Based Retrosynthesis Prediction

“…Instead, they construct models to predict reactants from products directly. Translation-based methods [ 6 , 11 , 22 , 23 ] use SMILES to represent molecules and treat the problem as a sequence-to-sequence task. MEGAN [ 8 ] treats the retrosynthesis problem as a graph transformation task, and trains the model to predict a sequence of graph edits that can transform the product into the reactants.…”

Multi-granularity scenarios understanding network for trajectory prediction

“…Transformers Transformer is a deep learning architecture proposed by Google in 2017. It has achieved great success in the field of NLP [44][45][46]. Due to the unique attention mechanism and the excellent performance in the field of NLP, researchers have great interest in its application in trajectory prediction.…”