Comparison of the Predictive Performance and Interpretability of Random Forest and Linear Models on Benchmark Data Sets

Robinson, Richard L. Marchese; Palczewska, Anna; Palczewski, Jan; Kidley, Nathan

doi:10.1021/acs.jcim.6b00753

Cited by 98 publications

(64 citation statements)

References 69 publications

(239 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, they noticed that their data sets were SBVS compliant and compared advantageously to the biased DUD sets, leading to a potential broader use of their sets. MUV sets were applied to the evaluation of VS tools (Tiikkainen et al, 2009 ; Abdo et al, 2010 ), the training of new QSAR models (Marchese Robinson et al, 2017 ) or molecular graph convolutions (Kearnes et al, 2016 ).…”

Section: Discussion and Recommendationsmentioning

confidence: 99%

Decoys Selection in Benchmarking Datasets: Overview and Perspectives

et al. 2018

View full text Add to dashboard Cite

Virtual Screening (VS) is designed to prospectively help identifying potential hits, i.e., compounds capable of interacting with a given target and potentially modulate its activity, out of large compound collections. Among the variety of methodologies, it is crucial to select the protocol that is the most adapted to the query/target system under study and that yields the most reliable output. To this aim, the performance of VS methods is commonly evaluated and compared by computing their ability to retrieve active compounds in benchmarking datasets. The benchmarking datasets contain a subset of known active compounds together with a subset of decoys, i.e., assumed non-active molecules. The composition of both the active and the decoy compounds subsets is critical to limit the biases in the evaluation of the VS methods. In this review, we focus on the selection of decoy compounds that has considerably changed over the years, from randomly selected compounds to highly customized or experimentally validated negative compounds. We first outline the evolution of decoys selection in benchmarking databases as well as current benchmarking databases that tend to minimize the introduction of biases, and secondly, we propose recommendations for the selection and the design of benchmarking datasets.

show abstract

Section: Discussion and Recommendationsmentioning

confidence: 99%

Decoys Selection in Benchmarking Datasets: Overview and Perspectives

et al. 2018

View full text Add to dashboard Cite

show abstract

“…While making machine learning models and predictions more accessible is important to demonstrate impact, efforts to increase the interpretability of these models beyond the "black box" are critical 63 . We and others have taken different routes to improve this aspect including tools to highlight contributions of models to test molecules 64 , identifying training compounds in the same neighborhood as test molecules and scores of model applicability or overlap 46,63 .…”

Section: Making Models More Accessible and Interpretablementioning

confidence: 99%

Exploiting machine learning for end-to-end drug discovery and development

et al. 2019

View full text Add to dashboard Cite

A variety of machine learning methods such as Naïve Bayesian, support vector machines and more recently deep neural networks are demonstrating their utility for drug discovery and development. These leverage the generally bigger data sets created from high throughput screening data and allow prediction of bioactivities for targets and molecular properties with increased levels of accuracy. We have only just begun to exploit the potential of these techniques but they may already be fundamentally changing the research process for identifying new molecules and/or repurposing old drugs. The integrated application of such machine learning models for end-to-end (E2E) application is broadly relevant and has considerable implications for developing future therapies and their targeting. Learning from history 'Those who do not remember the past are condemned to repeat it' (Santayana). This observation applies as much to drug discovery as it does to other aspects of human endeavor 1. The history of drug discovery is a prelude to the emerging potential of computerassisted data exploration. One constant in drug discovery is that every few years the estimated cost to develop drugs rises further. Less than 20 years ago, developing a drug took ~12 years, cost under a billion dollars, and the biggest challenges were failures due to efficacy or toxicity-induced attrition 2. in vitro pharmacological profiling implemented earlier in the drug discovery process helped to identify some predictable undesirable off-*

show abstract

“…Our previously published studies reported a better performance while using RF for classification tasks (46,47). Furthermore, RF is a commonly used algorithm which reported higher predictive ability compared to other ML-based algorithms (48,49). Thus, RF classifier was selected in our proposed study to risk stratify the patients.…”

Section: Supplementarymentioning

confidence: 99%

A low-cost machine learning-based cardiovascular/stroke risk assessment system: integration of conventional factors with image phenotypes

Jamthikar¹,

Gupta²,

Khanna³

et al. 2019

Cardiovasc. Diagn. Ther.

View full text Add to dashboard Cite

Background: Most cardiovascular (CV)/stroke risk calculators using the integration of carotid ultrasound image-based phenotypes (CUSIP) with conventional risk factors (CRF) have shown improved risk stratification compared with either method. However such approaches have not yet leveraged the potential of machine learning (ML). Most intelligent ML strategies use follow-ups for the endpoints but are costly and time-intensive. We introduce an integrated ML system using stenosis as an endpoint for training and determine whether such a system can lead to superior performance compared with the conventional ML system. Methods: The ML-based algorithm consists of an offline and online system. The offline system extracts 47 features which comprised of 13 CRF and 34 CUSIP. Principal component analysis (PCA) was used to select the most significant features. These offline features were then trained using the event-equivalent gold standard (consisting of percentage stenosis) using a random forest (RF) classifier framework to generate training coefficients. The online system then transforms the PCA-based test features using offline trained coefficients to predict the risk labels on test subjects. The above ML system determines the area under the curve (AUC) using a 10-fold cross-validation paradigm. The above system so-called "AtheroRisk-Integrated" was compared against "AtheroRisk-Conventional", where only 13 CRF were considered in a feature set. Results: Left and right common carotid arteries of 202 Japanese patients (Toho University, Japan) were retrospectively examined to obtain 395 ultrasound scans. AtheroRisk-Integrated system [AUC =0.80, P<0.0001, 95% confidence interval (CI): 0.77 to 0.84] showed an improvement of ~18% against AtheroRisk-Conventional ML (AUC =0.68, P<0.0001, 95% CI: 0.64 to 0.72). Conclusions: ML-based integrated model with the event-equivalent gold standard as percentage stenosis is powerful and offers low cost and high performance CV/stroke risk assessment.

show abstract

Comparison of the Predictive Performance and Interpretability of Random Forest and Linear Models on Benchmark Data Sets

Cited by 98 publications

References 69 publications

Decoys Selection in Benchmarking Datasets: Overview and Perspectives

Decoys Selection in Benchmarking Datasets: Overview and Perspectives

Exploiting machine learning for end-to-end drug discovery and development

A low-cost machine learning-based cardiovascular/stroke risk assessment system: integration of conventional factors with image phenotypes

Contact Info

Product

Resources

About