Transformer-CNN: Swiss knife for QSAR modeling and interpretation

Karpov, Pavel; Godin, Guillaume; Tetko, Igor V.

doi:10.1186/s13321-020-00423-w

Cited by 147 publications

(131 citation statements)

References 47 publications

Supporting

Mentioning

130

Contrasting

Order By: Relevance

“…Training the model to learn different representations of the same reaction by distorting the initial canonical data eliminated the effect of memorization and increased the generalization performance of models. These ideas are intensively used, e.g., for image recognition 39 , and have been already successfully used in the context of several chemical problems 27 – 30 , including reaction predictions 18 , 31 , but were limited to the input data. For the first time we showed that application of augmentation to the target data significantly boosts the quality of the reaction prediction.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

State-of-the-art augmented NLP transformer models for direct and single-step retrosynthesis

et al. 2020

Self Cite

View full text Add to dashboard Cite

We investigated the effect of different training scenarios on predicting the (retro)synthesis of chemical compounds using text-like representation of chemical reactions (SMILES) and Natural Language Processing (NLP) neural network Transformer architecture. We showed that data augmentation, which is a powerful method used in image processing, eliminated the effect of data memorization by neural networks and improved their performance for prediction of new sequences. This effect was observed when augmentation was used simultaneously for input and the target data simultaneously. The top-5 accuracy was 84.8% for the prediction of the largest fragment (thus identifying principal transformation for classical retro-synthesis) for the USPTO-50k test dataset, and was achieved by a combination of SMILES augmentation and a beam search algorithm. The same approach provided significantly better results for the prediction of direct reactions from the single-step USPTO-MIT test set. Our model achieved 90.6% top-1 and 96.1% top-5 accuracy for its challenging mixed set and 97% top-5 accuracy for the USPTO-MIT separated set. It also significantly improved results for USPTO-full set single-step retrosynthesis for both top-1 and top-10 accuracies. The appearance frequency of the most abundantly generated SMILES was well correlated with the prediction outcome and can be used as a measure of the quality of reaction prediction.

show abstract

Section: Discussionmentioning

confidence: 99%

“…The SMILES representation of molecules is ambiguous. Though the canonicalization procedure exists 26 , it has been shown that models benefit from using a batch of random SMILES (augmentation) during training and inference 27 – 30 . Recently, such augmentation was also applied to reaction modeling 11 , 18 , 31 , 32 .…”

Section: Introductionmentioning

confidence: 99%

State-of-the-art augmented NLP transformer models for direct and single-step retrosynthesis

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…Augmentation was done offline prior to training the network. Randomized SMILES were generated using RDKit by setting option doRandom = True, which was recently introduced to improve regression and classification models for physico-chemical properties [22,23]. As expected, the augmentation improved the percentage of generated valid SMILES while lowering the number of training epochs.…”

Section: Table 2 Comparison Architectures a B C And Dmentioning

confidence: 81%

“…Our calculations based on the publicly available dataset PubChem [21], clearly demonstrate that the use of bidirectional layers systematically improves the capability of the GEN to generate a vast set of new SMILES within the property space of the training set. Following excellent results of SMILES augmentation for smaller datasets to predict physicochemical properties [22][23][24] and generators [25], we have used SMILES augmentation to increase both the number and diversity of SMILES in the training set.…”

Section: Introductionmentioning

confidence: 99%

GEN: highly efficient SMILES explorer using autodidactic generative examination networks

et al. 2020

Self Cite

View full text Add to dashboard Cite

Recurrent neural networks have been widely used to generate millions of de novo molecules in defined chemical spaces. Reported deep generative models are exclusively based on LSTM and/or GRU units and frequently trained using canonical SMILES. In this study, we introduce Generative Examination Networks (GEN) as a new approach to train deep generative networks for SMILES generation. In our GENs, we have used an architecture based on multiple concatenated bidirectional RNN units to enhance the validity of generated SMILES. GENs autonomously learn the target space in a few epochs and are stopped early using an independent online examination mechanism, measuring the quality of the generated set. Herein we have used online statistical quality control (SQC) on the percentage of valid molecular SMILES as examination measure to select the earliest available stable model weights. Very high levels of valid SMILES (95-98%) can be generated using multiple parallel encoding layers in combination with SMILES augmentation using unrestricted SMILES randomization. Our trained models combine an excellent novelty rate (85-90%) while generating SMILES with strong conservation of the property space (95-99%). In GENs, both the generative network and the examination mechanism are open to other architectures and quality criteria.

show abstract

“…When making use of feature attribution approaches, it is advisable to choose comprehensible molecular descriptors or representations for model construction (Box 2). Recently, architectures borrowed from the natural language processing field, such as long short-term memory networks 76 and transformers 77 , have been used as feature attribution techniques to identify portions of simplified molecular input line entry systems (SMILES) 78 strings that are relevant for bioactivity or physicochemical properties 79,80 . These approaches constitute a first attempt to bridge the gap between the deep learning and medicinal chemistry communities, by relying on representations (atom and bond types, and molecular connectivity 78 ) that bear direct chemical meaning and need no posterior descriptor-to-molecule decoding.…”

Section: Relevance Of Input Featuresmentioning

confidence: 99%

Drug discovery with explainable artificial intelligence

2020

View full text Add to dashboard Cite

arious concepts of 'artificial intelligence' (AI) have been successfully adopted for computer-assisted drug discovery in the past few years 1-3. This advance is mostly owed to the ability of deep learning algorithms, that is, artificial neural networks with multiple processing layers, to model complex nonlinear inputoutput relationships, and perform pattern recognition and feature extraction from low-level data representations. Certain deep learning models have been shown to match or even exceed the performance of the familiar existing machine learning and quantitative structure-activity relationship (QSAR) methods for drug discovery 4-6. Moreover, deep learning has boosted the potential and broadened the applicability of computer-assisted discovery, for example, in molecular design 7,8 , chemical synthesis planning 9,10 , protein structure prediction 11 and macromolecular target identification 12,13. The ability to capture intricate nonlinear relationships between input data (for example, chemical structure representations) and the associated output (for example, assay readout) often comes at the price of limited comprehensibility of the resulting model. While there have been efforts to explain QSARs in terms of algorithmic insights and molecular descriptor analysis 14-19 , deep neural network models notoriously elude immediate accessibility by the human mind 20. In medicinal chemistry in particular, the availability of 'rules of thumb' correlating biological effects with physicochemical properties underscores the willingness, in certain situations, to sacrifice accuracy in favour of models that better fit human intuition 21-23. Thus, blurring the lines between the 'two QSARs' 24 (that is, mechanistically interpretable versus highly accurate models) may be key to accelerated drug discovery with AI 25. Automated analysis of medical and chemical knowledge to extract and represent features in a human-intelligible format dates back to the 1990s 26,27 , but has been receiving increasing attention due to the re-emergence of neural networks in chemistry and healthcare. Given the current pace of AI in drug discovery and related fields, there will be an increased demand for methods that help us understand and interpret the underlying models. In an effort to mitigate the lack of interpretability of certain machine learning models, and to augment human reasoning and decision-making, 28 , attention has been drawn to explainable AI (XAI) approaches 29,30. Providing informative explanations alongside the mathematical models aims to (1) render the underlying decision-making process transparent ('understandable') 31 , (2) avoid correct predictions for the wrong reasons (the so-called clever Hans effect) 32 , (3) avert unfair biases or unethical discrimination 33 and (4) bridge the gap between the machine learning community and other scientific disciplines. In addition, effective XAI can help scientists navigate 'cognitive valleys' 28 , allowing them to hone their knowledge and beliefs on the investigated process 34. While the e...

show abstract

Transformer-CNN: Swiss knife for QSAR modeling and interpretation

Cited by 147 publications

References 47 publications

State-of-the-art augmented NLP transformer models for direct and single-step retrosynthesis

State-of-the-art augmented NLP transformer models for direct and single-step retrosynthesis

GEN: highly efficient SMILES explorer using autodidactic generative examination networks

Drug discovery with explainable artificial intelligence

Contact Info

Product

Resources

About