A cross-study analysis of drug response prediction in cancer cell lines

Abstract

To enable personalized cancer treatment, machine learning models have been developed to predict drug response as a function of tumor and drug features. However, most algorithm development efforts have relied on cross-validation within a single study to assess model accuracy. While an essential first step, cross-validation within a biological data set typically provides an overly optimistic estimate of the prediction performance on independent test sets. To provide a more rigorous assessment of model generalizability between different studies, we use machine learning to analyze five publicly available cell line-based data sets: National Cancer Institute 60, ancer Therapeutics Response Portal (CTRP), Genomics of Drug Sensitivity in Cancer, Cancer Cell Line Encyclopedia and Genentech Cell Line Screening Initiative (gCSI). Based on observed experimental variability across studies, we explore estimates of prediction upper bounds. We report performance results of a variety of machine learning models, with a multitasking deep neural network achieving the best cross-study generalizability. By multiple measures, models trained on CTRP yield the most accurate predictions on the remaining testing data, and gCSI is the most predictable among the cell line data sets included in this study. With these experiments and further simulations on partial data, two lessons emerge: (1) differences in viability assays can limit model generalizability across studies and (2) drug diversity, more than tumor diversity, is crucial for raising model generalizability in preclinical screening.

A systematic evaluation of deep learning methods for the prediction of drug synergy in cancer

“…In addition, the models were validated with data from within the same drug combination screening study. Although this is a common model evaluation strategy, cross-validation within a single study has been shown to overestimate model generalizability [73]. In the future it would be interesting to assess different DL modeling strategies by performing a cross-study evaluation of the models as well [73].…”

“…Although this is a common model evaluation strategy, cross-validation within a single study has been shown to overestimate model generalizability [73]. In the future it would be interesting to assess different DL modeling strategies by performing a cross-study evaluation of the models as well [73]. Since ComboScore predictions should be independent of the order of the drugs in a given combination, we considered reverse drug order < cell line, drugB, drugA > triplets to be duplicates and the ComboScores for experiments involving the same cell line and same drug combination were averaged.…”

Data augmentation and multimodal learning for predicting drug response in patient-derived xenografts from gene expressions and histology images

“…The sample size of PDXs is usually orders of magnitude smaller than the analogous CCL datasets. It has been shown that increasing the amount of training samples improves generalization performance of supervised learning models in vision and text applications [17,18], as well as drug response models in CCLs [19,20]. Collecting PDX response data, either through experiments or integration of multiple datasets, carries considerable technical and financial challenges.…”

HPTMT Parallel Operators for High Performance Data Science and Data Engineering

“…UNOMT application is part of CANDLE Wozniak et al (2020), Xia et al (2021) research conducted by Argonne National Laboratory, focusing on automated detection of tumour cells using a deep learning approach. The uniqueness of this approach is the composition of a data engineering workload followed by a deep learning workload written in PyTorch.…”