Assessing QSAR Limitations - A Regulatory Perspective

Abstract

Wider acceptance of QSARs would result in a constellation of benefits and savings to both private and public sectors. For this to occur, particularly in regulatory applications, a model's limitations need to be identified. We define a model's limitations as encompassing assessment of overall prediction accuracy, applicability domain and chance correlation. A general guideline is presented in this review for assessing a model's limitations with emphasis on and examples of application with consensus modeling methods. More specifically, we discuss the commonalities and differences between external validation and cross-validation for assessing a model's limitations. We illustrate two common ways of assessing overall prediction accuracy, depending on whether or not the intended application domain is predefined. Since even a high quality model will have different confidence in accuracy for predicting different chemicals, we further demonstrate using the novel Decision Forest consensus modeling method a means to determine prediction confidence (i.e., certainty for an individual chemical's prediction) and domain extrapolation (i.e., the prediction accuracy for a chemical that is outside the chemistry space defined by the training chemicals). We show that prediction confidence and domain extrapolation are related measures that together determine the applicability domain of a model, and that prediction confidence is the more important measure. Lastly, the importance of assessing chance correlation is emphasized, and illustrated with several examples of models having a high degree of chance correlations despite cross-validation indicating high prediction accuracy. Generally, a dataset with a skewed distribution, small data size and/or low signal/noise ratio tends to produce a model with high chance correlation.We conclude that it is imperative to assess all three aspects (i.e., overall accuracy, applicability domain and chance correlation) of a model for the regulatory acceptance of QSARs.

On further application of r as a metric for validation of QSAR models

“…Validation of models is a hot topic in the field of quantitative structure-activity relationships (QSARs) which have important applications in drug discovery research, environmental fate modeling, property prediction, etc [1][2][3][4][5][6][7]. One of the important objectives of QSAR modeling is to predict activity/property/ toxicity of new compounds falling within the domain of applicability of the developed models.…”

“…One of the important objectives of QSAR modeling is to predict activity/property/ toxicity of new compounds falling within the domain of applicability of the developed models. As QSARs are being used for regulatory decisions [2], reliability of the models and confidence in their predictions are very important aspects, which are judged during the validation process. This has recently created much interest in defining validation criteria for the acceptance of QSAR models [8][9][10].…”

On Selection of Training and Test Sets for the Development of Predictive QSAR models

“…Quantitative Structure -Activity Relationship (QSAR) methods have been applied widely in the pharmaceutical industry for drug discovery, lead optimization, risk assessment, toxicity prediction and regulatory decisions [1]. The QSAR models are useful for various purposes including the prediction of activities of untested chemicals.…”

Engineered Nanomaterials in Food: Implications for Food Safety and Consumer Health

“…QSARs are theoretical models that relate the structure or physicochemical properties of substances to their biological activities. QSARs are being applied in many disciplines, for example in risk assessment, and toxicity prediction [208], as well as in drug discovery and lead optimization [209], but they have yet few applications for NPs (nano-QSAR) [210]. Recently nano-QSAR was applied to predict the cytotoxicity of metal oxide NPs [211], as well as predictive models of cellular uptake and apoptosis induced by NPs for several cell types was developed by Epa et al .…”