Benchmarks for interpretation of QSAR models

Abstract

Interpretation of QSAR models is useful to understand the complex nature of biological or physicochemical processes, guide structural optimization or perform knowledge-based validation of QSAR models. Highly predictive models are usually complex and their interpretation is non-trivial. This is particularly true for modern neural networks. Various approaches to interpretation of these models exist. However, it is difficult to evaluate and compare performance and applicability of these ever-emerging methods. Herein, we developed several benchmark data sets with end-points determined by pre-defined patterns. These data sets are purposed for evaluation of the ability of interpretation approaches to retrieve these patterns. They represent tasks with different complexity levels: from simple atom-based additive properties to pharmacophore hypotheses. We proposed several quantitative metrics of interpretation performance. Applicability of benchmarks and metrics was demonstrated on a set of conventional models and end-to-end graph convolutional neural networks interpreted by the previously suggested universal ML-agnostic approach for structural interpretation. We anticipate these benchmarks to be useful in evaluation of new interpretation approaches and investigation of decision making of complex “black box” models.

Benchmarking Molecular Feature Attribution Methods with Activity Cliffs

“…In the specific context of chemoinformatics, several attempts have been made in recent years with the aim of uncovering black-box ML algorithms in property prediction tasks. [13][14][15] In particular, while some studies go to great lengths to show how several modern feature attribution methods can be used to some extent to identify structural motifs, 16,17 or property cliffs, 18 it is hard to evaluate which feature attribution methods work best and under which specific conditions. Along these lines, a study by Sánchez-Lengeling et al 19 proposed a quantitative benchmark for several well-known feature attribution techniques in conjunction with GNNs.…”

Ligandformer: A Graph Neural Network for Predicting Compound Property with Robust Interpretation

“…For molecule chemical representation, current GNN methods [9,11,8] usually process 2D graph as description of natural chemical graph, in which nodes represent atoms integrating different chemical attributes, and edges represent bonds connecting atoms to one another. There are mainly three advantages of using 2D graph description: (1) graph preserves clear and stable information of chemical structure, (2) it represents invariant molecule regardless of entry position in line notation (e.g., SMILES [18]), (3) it can be easily computed and optimized by GNN methods.…”

“…However, many DL models function as black-boxes [7], which means that given a molecule with physiochemical/biological and structural features, DL models usually predict a simple global score for certain desired property, leaving the inference rationales, such like local judgement on chemical structure, an unknown status. Interpretation of such rationales is useful to reveal relationship between structure and property, optimize compound structure, and validate DL models with subjective opinion (chemical or biological knowledge) [9,11,8]. Especially, recent Graph Neural Network (GNN) based methods [1,6,12,13,14,15] were wildly designed and applied as QSAR models for predicting compound properties.…”

Improving QSAR Modeling for Predictive Toxicology using Publicly Aggregated Semantic Graph Data and Graph Neural Networks

“…However, models based on fingerprints are challenging to interpret. 7,21 Although each field of a MACCS fingerprint corresponds to meaningful chemical properties (such as whether the chemical contains multiple aromatic rings, or at least one nitrogen atom), the fingerprint is largely inscrutable in QSAR applications, since biological activity is the result of many higher-order interactions between the chemical of interest and biomolecules.…”