Understanding the Sources of Performance in Deep Learning Drug Response Prediction Models

Abstract

MotivationAnti-cancer drug response prediction (DRP) using cancer cell lines plays a vital role in stratified medicine and drug discovery. Recently there has been a surge of new deep learning (DL) models for DRP that show promising performance improvements. However, different models use different input data modalities and neural network architectures making it hard to find the source of these improvements.ResultsWe consider three DL DRP models with reported state-of-the-art performance and use genomics or transcriptomics profiles of cell lines and chemical profiles of drugs as input data. We also construct null hypothesis benchmarks that simply leverage patterns in truth values and thus do not use omics or chemical profiles. By experimenting with these DL models and comparing them with our null hypothesis benchmarks we show that for multiple testing types, much or all of the current reported performance is a property of the training truth values. Furthermore, we show that for cancer blind testing (testing on unseen cell lines) no performance comes from drug features, instead, performance is due to the transcriptomics cell line profiles.