Decision threshold adjustment in class prediction

Abstract

Standard classification algorithms are generally designed to maximize the number of correct predictions (concordance). The criterion of maximizing the concordance may not be appropriate in certain applications. In practice, some applications may emphasize high sensitivity (e.g., clinical diagnostic tests) and others may emphasize high specificity (e.g., epidemiology screening studies). This paper considers effects of the decision threshold on sensitivity, specificity, and concordance for four classification methods: logistic regression, classification tree, Fisher's linear discriminant analysis, and a weighted k-nearest neighbor. We investigated the use of decision threshold adjustment to improve performance of either sensitivity or specificity of a classifier under specific conditions. We conducted a Monte Carlo simulation showing that as the decision threshold increases, the sensitivity decreases and the specificity increases; but, the concordance values in an interval around the maximum concordance are similar. For specified sensitivity and specificity levels, an optimal decision threshold might be determined in an interval around the maximum concordance that meets the specified requirement. Three example data sets were analyzed for illustrations.

A comparison of selected parametric and imputation methods for estimating snag density and snag quality attributes

“…Sensitivity and specificity can depend on the prevalence of large snags, because classification methods tend to favor the larger class, when the size of presence and absence classes is unequal (Chen et al, 2006). When the decision threshold s = 0.5, we found that sensitivity was high on public lands where prevalence of large snags was high, but low on private lands where prevalence of large snags was small, thus favoring the larger presence class.…”

“…Adjusting s is fairly easy for logistic regression models that predict probabilities. If three or more nearest neighbors are used in the RF approach, s can also be adjusted (see Chen et al, 2006). The logistic regression model provided higher accuracies for the classification of large snag presence/absence than the RF approach.…”

“…The RF imputation method (Crookston and Finley, 2008) was employed to impute a single response: absence or presence of large snags. NN methods such as RF, when used to produce a binary outcome of either 0 or 1 as in the given case, cannot be used in combination with a threshold adjustment that takes prevalence of large snags into account (Chen et al, 2006). In order to apply an adjusted decision threshold s to the RF imputation approach, we used a simple average of the values observed for the five nearest neighbors to determine presence or absence of large snags.…”

“…In order to classify a predicted probability (continuous between 0 and 1) as either absence or presence of large snags, a decision threshold s needs to be chosen. The decision threshold s was chosen to reflect the prevalence of large snags (proportion of plots in which large snags are present), which results in balancing the values of sensitivity (proportion of correctly classified presences) and specificity (proportion of correctly classified absences) (Chen et al, 2006).…”

Mapping change of older forest with nearest-neighbor imputation and Landsat time-series

“…The classification accuracy for older forest (vs. not older forest) will vary with the threshold value applied, by differentially affecting the sensitivity and specificity of the spatial predictions, as demonstrated by Chen et al (2006). With imputed maps, the overall accuracy, sensitivity, and specificity are readily post-calculated for any threshold value of interest (see Moeur et al (2011), Table 9, for one example), and map users may wish to consider these error rates in choosing a threshold value.…”