Benchmarks for interpretation of QSAR models

Abstract

Interpretation of QSAR models is useful to understand the complex nature of biological or physicochemical processes, guide structural optimization or perform knowledge-based validation of QSAR models. Highly predictive models are usually complex and their interpretation is non-trivial. This is particularly true for modern neural networks. Various approaches to interpretation of these models exist. However, it is difficult to evaluate and compare performance and applicability of these ever-emerging methods. Herein, we developed several benchmark data sets with end-points determined by pre-defined patterns. These data sets are purposed for evaluation of the ability of interpretation approaches to retrieve these patterns. They represent tasks with different complexity levels: from simple atom-based additive properties to pharmacophore hypothesis. We proposed several quantitative metrics of interpretation performance. Applicability of benchmarks and metrics was demonstrated on a set of conventional models and end-to-end graph convolutional neural networks, interpreted by the previously suggested universal ML-agnostic approach for structural interpretation. We anticipate these benchmarks to be useful in evaluation of new interpretation approaches and investigation of decision making of complex “black box” models.

Explainable Machine Learning for Property Predictions in Compound Optimization

“…Moreover, model explanation methods are challenging to benchmark, given the sparsity of generally accepted standards in the field. Despite recent efforts to generate meaningful benchmark data sets, 43 the field still lacks clear benchmarks for model interpretability and additional work is required to rigorously compare explanation methods. Without doubt, proper evaluation of model interpretation methods will benefit from close collaboration between data scientists and medicinal chemists.…”

“…42 Of note, using similarity maps or the universal approach, GCNNs were more resistant to explanation than other ML algorithms. 43 Furthermore, methods based on attention mechanisms have been introduced to understand GCNN predictions (Figure 3). 44 Tang et al applied this approach to molecular lipophilicity and aqueous solubility predictions using message passing neural networks.…”

“…This adaptable representation can be perceived as further extending the concept of Morgan fingerprints (circular fingerprints with a given bond diameter) and might yield features more representative for a given modeling task . Of note, using similarity maps or the universal approach, GCNNs were more resistant to explanation than other ML algorithms . Furthermore, methods based on attention mechanisms have been introduced to understand GCNN predictions (Figure ).…”

“…Feature, atom, or fragment importance values should best be presented in an intuitive graphical manner to understand model decisions. Coloring a molecule according to estimated atom relevance provides an intuitive explanation for individual predictions, regardless of how importance values were estimated, ,, as illustrated in Figure A. Feature mapping to compound structures is sometimes required.…”

Deep Machine Learning for Computer-Aided Drug Design

“…Moreover, it will be of critical importance for the future of DL in CADD to concentrate on prospective applications with measurable impact on experimental programs. To these ends, it will also be crucial to reduce the black box character of DL by integrating methods for explaining predictions (Fisher et al, 2019;Murdoch et al, 2019;Lundberg et al, 2020;Matveieva and Polishchuk, 2021;Rodríguez-Pérez and Bajorath, 2021). The ability to rationalize DNN predictions will further increase the acceptance of DL for experimental design.…”