Transformer Performance for Chemical Reactions: Analysis of Different Predictive and Evaluation Scenarios

Abstract

The prediction of chemical reaction pathways has been accelerated by the development of novel machine learning architectures based on the deep learning paradigm. In this context, deep neural networks initially designed for language translation have been used to accurately predict a wide range of chemical reactions. Among models suited for the task of language translation, the recently introduced molecular transformer reached impressive performance in terms of forward-synthesis and retrosynthesis predictions. In this study, we first present an analysis of the performance of transformer models for product, reactant, and reagent prediction tasks under different scenarios of data availability and data augmentation. We find that the impact of data augmentation depends on the prediction task and on the metric used to evaluate the model performance. Second, we probe the contribution of different combinations of input formats, tokenization schemes, and embedding strategies to model performance. We find that less stable input settings generally lead to better performance. Lastly, we validate the superiority of round-trip accuracy over simpler evaluation metrics, such as top-k accuracy, using a committee of human experts and show a strong agreement for predictions that pass the round-trip test. This demonstrates the usefulness of more elaborate metrics in complex predictive scenarios and highlights the limitations of direct comparisons to a predefined database, which may include a limited number of chemical reaction pathways.

Do Chemformers Dream of Organic Matter? Evaluating a Transformer Model for Multistep Retrosynthesis

“…The forward Chemformer model was trained with reactants as input and was used for the evaluation of retrosynthesis predictions. A forward model usually takes both reagents and reactants as input since it yields a more accurate model . We, therefore, validated our forward model with one trained using both reactants and reagents as input, as shown in Figure S1.…”

“…For the multistep retrosynthesis analysis, we used a beam size of 10 in the round-trip forward Chemformer inference. Recent work has shown that round-trip accuracy may agree more with chemists’ judgments of correct reaction suggestions than exact matching accuracy . However, because the round-trip accuracy depends on the accuracy of the forward model and does not entail exact accuracy, these two metrics can be considered complementary.…”

“…It was pre-trained using SMILES to represent the “language” of chemical structures . Research investigating the use of similar transformer models for single-step retrosynthesis prediction, ,,− as well as forward prediction ,,,,,− and reagent prediction have demonstrated superior performance of these models compared to more classic approaches such as template-based models or graph neural networks . Since the first implementations of a retrosynthesis transformer model, , several enhancements and developments have been presented, including a diversity-enhanced transformer, triple-transformer validation loop, and disconnection-prompted transformer .…”

Leveraging Language Model Multitasking To Predict C–H Borylation Selectivity

“…Atoms with two-letter symbols, such as chlorine, are represented by two tokens rather than by one, which may be counterintuitive. However, in an independent study, the use of data-driven tokenization was shown not to bring about an improvement in molecular generation accuracy . We decided, therefore, to focus on testing the character-wise tokenizer.…”

SynRoute: A Retrosynthetic Planning Software

“…In that direction, it cannot be used to directly infer retrosynthetic reactions, but it can be used to verify the feasibility of reactions. On the other hand, the works of refs − use transformer neural networks to predict possible reactants, given a product, and can be used to create reactions retrosynthetically. However, these studies only measure the accuracy of a single step, that is, the accuracy of single reactions, and not as complete routes from purchasable compounds to target compounds.…”