ROC Method for the Evaluation of Multi-class Segmentation/Classification Algorithms with Infrared Imagery

Abstract

The classification of image regions of interest in an image is an important area of research. Generally most investigations concentrate on the optimisation of the constituent parts of the system without regard to the overall performance. This work takes a system centred approach. Using a novel multi-class receiver operating characteristic, which also allows for the inherent uncertainty present, it is shown that the influence of different region based segmentation algorithms on the performance of classification algorithms can be determined. The results generated, using this approach, for an airborne infrared application highlight the non-linear relationship between the constituent algorithms and show quantitatively that the system performance can be strongly class and segmenter/classifier dependent.

Speech recognition by indexing and sequencing

“…Since this kind of analysis is not significant for our application, we follow a more direct approach by computing the number of true positives (TP), true negatives (TN), false positives (FP) and false negatives (FN) on a per-class basis (50). Suppose that we have 3 classes and we want to compute the statistics for class 1; we would then regard the values in the confusion matrix as follows: The Real Time Factor (RTF), defined as the ratio of the time needed to process an input sequence to the duration of the sequence, was measured on a MacBook Pro with a 2.4 GHz Intel Core 2 Duo CPU and 4 GB RAM.…”

On the use of the overlapping area matrix for image segmentation evaluation: A survey and new performance measures

“…An empirical discrepancy method can still be more general since the evaluation is based on contrasting the algorithm's output with an already computed reference. The number of pixels incorrectly classified as edge pixels or the number of incorrectly segmented pixels, their position and the number of regions are among the different discrepancy measures proposed in the literature (Baddeley, 1992;Goumeidane et al, 2003;Heyden, 1989;Huang and Dom, 1995;Lewis and Brown, 2001;Lim and Lee, 1990;Odet et al, 2002;Pratt, 1978;Rees et al, 2002;RomanRoldan et al, 2001;Strasters and Gerbrands, 1991;Weszka and Rosenfeld, 1978;Yasnoff and Bacus, 1984;Yasnoff et al, 1977). Additional discrepancy measures based on region features such as area, eccentricity or perimeter, among others, have also been considered (Zhang, 1995;Gerbrands, 1992, 1994).…”

“…the classification threshold). It is therefore mainly suited for two-class classification problems, such as edge detection, although (Rees et al, 2002) addressed multiclass classification evaluation by means of ROC analysis. As can be observed, ROC analysis does not provide new performance measures, it is rather a change of philosophy, as Bowyer et al remarked regarding the evaluation of edge detectors (Bowyer et al, 2001, p. 80).…”

“…ROC analysis comes from psychophysics and signal detection theory and has received an important amount of attention within the vision community (Alexander and Buxton, 1997;Bowyer et al, 2001;Edwards et al, 1999;Forbes and Draper, 2000;Ramesh and Haralick, 1992;Rees et al, 2002;Shin et al, 2001;Southall et al, 2000;Yitzhaky and Peli, 2003). A ROC curve plots true positive rates against false positive rates as a decision parameter of the classifier is varied (i.e.…”