Interpretable deep learning architectures for improving drug response prediction performance: myth or reality?

Abstract

Motivation Interpretable deep learning (DL) models that can provide biological insights, in addition to accurate predictions, are of great interest to the biomedical community. Recently, interpretable DL models that incorporate signaling pathways have been proposed for drug response prediction. While these models improve interpretability, it is unclear whether this comes at the cost of less accurate drug response predictions, or a prediction improvement can also be obtained. Results We comprehensively and systematically assessed four state-of-the-art interpretable DL models using three pathway collections to assess their ability in making accurate predictions on unseen samples from the same dataset, as well as their generalizability to an independent dataset. Our results showed that models that explicitly incorporate pathway information in the form of a latent layer perform worse compared to models that incorporate this information implicitly. However, in most evaluation setups, the best performance was achieved using a black-box multi-layer perceptron, and the performance of a random forests baseline was comparable to those of the interpretable models. Replacing the signaling pathways with randomly generated pathways showed a comparable performance for the majority of the models. Finally, the performance of all models deteriorated when applied to an independent dataset. These results highlight the importance of systematic evaluation of newly proposed models using carefully selected baselines. We provide different evaluation setups and baseline models that can be used to achieve this goal. Availability Implemented models and datasets are provided in https://doi.org/10.5281/zenodo.7787178 and https://doi.org/10.5281/zenodo.7101665, respectively. Supplementary information Supplementary data are available at Bioinformatics online.

A comprehensive benchmarking of machine learning algorithms and dimensionality reduction methods for drug sensitivity prediction

“…In an attempt to make models more interpretable, biological knowledge is often explicitly encoded into prediction models, e.g. using pathway-layers in neural networks [ 4 , 35 , 67 ], exploiting known protein interactions [ 22 ] or encoding information on known markers of drug response [ 11 ]. Li et al .…”

“…Li et al . found, however, that the explicit incorporation of biological knowledge may decrease model performance and lead to false conclusions [ 35 ]. Hence, the assumptions that are introduced by adding biological knowledge to a model should be carefully investigated.…”

“…However, Li et al . [ 35 ] found that many deep learning approaches are not superior in performance to simple feed-forward neural networks or random forests for drug sensitivity prediction. Apart from that, keeping models as simple and interpretable as possible is desirable for studying the relation between cellular features and drug response and creating trustworthy, comprehensible predictions to enable ML use for clinical decision support.…”

Machine Learning and Artificial Intelligence in drug repurposing – challenges and perspectives

“…Second, existing methods rarely compare themselves against simpler baseline models, making the superiority of highly complex models questionable. Third, it can be shown that even a simple mean predictor, which just reports mean drug response in the training data, can achieve surprisingly good performance, questioning whether current methods actually learn cell-line specific drug response as advertised and whether current performance measures are adequate [63]. Indeed, it can be shown that the current practice of reporting the correlation between predicted and observed drug response across all drugs is subject to Simpson's paradox (Figure 2) due to drug-specific average responses.…”

Understanding the Sources of Performance in Deep Learning Drug Response Prediction Models

“…None of these benchmarks use any omics data or drug structures. We note that the drug and CL average benchmarks have recently been shown to have strong performance (Branson et al, 2023;Li et al, 2023) Where n Dtrainc is the number of drug cell line pairs in the training set that include c.…”