Getting past diversity in assessing virtual library designs

Clark, Robert D.

doi:10.1590/s0103-50532002000600009

Cited by 4 publications

(4 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Representation in the full data set is biased, so such simple random subsets are biased as well. The results obtained using that training set (set R ) are included here to illustrate the effect of sampling bias due to structural redundancy [ 39 , 44 , 45 ].…”

Section: Resultsmentioning

confidence: 99%

DPRESS: Localizing estimates of predictive uncertainty

Clark

2009

J Cheminform

View full text Add to dashboard Cite

BackgroundThe need to have a quantitative estimate of the uncertainty of prediction for QSAR models is steadily increasing, in part because such predictions are being widely distributed as tabulated values disconnected from the models used to generate them. Classical statistical theory assumes that the error in the population being modeled is independent and identically distributed (IID), but this is often not actually the case. Such inhomogeneous error (heteroskedasticity) can be addressed by providing an individualized estimate of predictive uncertainty for each particular new object u: the standard error of prediction su can be estimated as the non-cross-validated error st* for the closest object t* in the training set adjusted for its separation d from u in the descriptor space relative to the size of the training set.The predictive uncertainty factor γt* is obtained by distributing the internal predictive error sum of squares across objects in the training set based on the distances between them, hence the acronym: Distributed PRedictive Error Sum of Squares (DPRESS). Note that st* and γt*are characteristic of each training set compound contributing to the model of interest.ResultsThe method was applied to partial least-squares models built using 2D (molecular hologram) or 3D (molecular field) descriptors applied to mid-sized training sets (N = 75) drawn from a large (N = 304), well-characterized pool of cyclooxygenase inhibitors. The observed variation in predictive error for the external 229 compound test sets was compared with the uncertainty estimates from DPRESS. Good qualitative and quantitative agreement was seen between the distributions of predictive error observed and those predicted using DPRESS. Inclusion of the distance-dependent term was essential to getting good agreement between the estimated uncertainties and the observed distributions of predictive error. The uncertainty estimates derived by DPRESS were conservative even when the training set was biased, but not excessively so.ConclusionDPRESS is a straightforward and powerful way to reliably estimate individual predictive uncertainties for compounds outside the training set based on their distance to the training set and the internal predictive uncertainty associated with its nearest neighbor in that set. It represents a sample-based, a posteriori approach to defining applicability domains in terms of localized uncertainty.

show abstract

Section: Resultsmentioning

confidence: 99%

DPRESS: Localizing estimates of predictive uncertainty

Clark

2009

J Cheminform

View full text Add to dashboard Cite

show abstract

“…Those strategies, in turn, reflect fashions and trends in pharmaceutical development. Any such data set tends to be structurally 'clumpy' [10][11][12] and the bias in sampling represented by that clumpiness compromises many of the otherwise excellent statistical properties of an ROC analysis carried out on truly independent observations (www.interscience.wiley.com) DOI: 10.1002/cem.1240…”

Section: Introductionmentioning

confidence: 99%

The effect of structural redundancy in validation sets on virtual screening performance

Clark

Shepphird²,

Holliday

2009

Journal of Chemometrics

Self Cite

View full text Add to dashboard Cite

The performance of a classification model is often assessed in terms of how well it separates a set of known observations into appropriate classes. If the validation sets used for such analyses are redundant due to bias in sampling, the relevance of the conclusions drawn to prospective work in which new kinds of positives are sought may be compromised. In the case of the various virtual screening techniques used in modern drug discovery, such bias generally appears as over‐representation of particular structural subclasses in the test set. We show how clustering by substructural similarity, followed by applying arithmetic and harmonic weighting schemes to receiver operating characteristic (ROC) curves, can be used to identify validation sets that are biased due to such redundancies. This can be accomplished qualitatively by direct examination or quantitatively by comparing the areas under the respective linear or semilog curves (AUCs or pAUCs). Copyright © 2009 John Wiley & Sons, Ltd.

show abstract

“…This limitation has been alleviated by pivoting between iterations, so that choosing candidates for subsamples alternates between reagent pools. This allows the efficient selection of multiblock sublibraries from full combinatorials. − Here, that logic has been further extended to allow generation of sparse blocks and by incorporating bias into the system so as to shift reagent selection in favor of one or another subclass of reagents.…”

Section: Introductionmentioning

confidence: 99%

“…This allows the efficient selection of multiblock sublibraries from full combinatorials. [4][5][6] Here, that logic has been further extended to allow generation of sparse blocks and by incorporating bias into the system so as to shift reagent selection in favor of one or another subclass of reagents.…”

Section: Introductionmentioning

confidence: 99%

OptDesign: Extending Optimizable k-Dissimilarity Selection to Combinatorial Library Design

Clark¹,

Kar²,

Akella³

et al. 2003

J. Chem. Inf. Comput. Sci.

Self Cite

View full text Add to dashboard Cite

Optimizable k-dissimilarity (OptiSim) selection entails drawing a series of subsamples of size k from a population and choosing the "best" candidate from each such subsample for inclusion in the selection set. By varying the size of the subsample, one can control the balance between representativeness and diversity in the selection set obtained. In the original formulation, a uniform random sampling from among valid candidates was used to draw the subsamples from a single target population. Here we describe in detail two key modifications that serve to extend the OptiSim methodology to vector selection for interdependent variables, specifically as applied to the design of combinatorial sublibraries. The first modification involves pivoting between variables: subsamples are drawn from each reagent pool in turn, with the viability of each candidate being evaluated in isolation as well as in terms of the products it will produce from complementary reagents already selected. The filters applied may be static or dynamic in nature, with molecular weight and hydrophobicity being examples of the former and structural diversity with respect to reagents already selected being an example of the latter. The second key modification is adding the ability to bias the selection of candidate reagents for inclusion in the subsamples. Taken together, these modifications support the efficient generation of multiblock and other sparse matrix designs that are both representative and diverse, and for which "backfilling" of designs edited to remove undesirable reagents or products is straightforward. The method is intrinsically fast and efficient, since enumeration of the full combinatorial is not required- only those candidates actually considered for inclusion need be evaluated. Moreover, because the subsample selection step is separate from the diversity-based selection of the "best" candidate, incorporating such bias in favor of a competing criterion such as low price provides a "natural," nonparametric mechanism for generating designs that are likely to be "good" in a double-objective, Pareto sense.

show abstract

Getting past diversity in assessing virtual library designs

Cited by 4 publications

References 9 publications

DPRESS: Localizing estimates of predictive uncertainty

DPRESS: Localizing estimates of predictive uncertainty

The effect of structural redundancy in validation sets on virtual screening performance

OptDesign: Extending Optimizable k-Dissimilarity Selection to Combinatorial Library Design

Contact Info

Product

Resources

About