On deflection as a performance criterion in detection

Picinbono, B.

doi:10.1109/7.395235

Cited by 120 publications

(83 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are other measures of performance such as output signal to noise ratio, or de ection criterion (36) , but the AUC seems to be the only one that is independent of the decision threshold and not biased by prior probabilities.…”

Section: The Standard Error Of Aucmentioning

confidence: 99%

The use of the area under the ROC curve in the evaluation of machine learning algorithms

Bradley

1997

Pattern Recognition

5,593

2,981

View full text Add to dashboard Cite

SummaryIn this paper we i n vestigate the use of receiver operating characteristic (ROC) curve f o r the evaluation of machine learning algorithms. In particular, we i n vestigate the use of the area under the ROC curve ( A UC) as a measure of classi er performance. The machine learning algorithms used are chosen to be representative of those in common use: two decision trees (C4.5 and Multiscale Classi er) two n e u r a l n e t works (Perceptron and Multi-layer Perceptron) and two statistical methods (K-Nearest Neighbours and a Quadratic Discriminant F unction).The evaluation is done using six, \real world," medical diagnostics data sets that contain a varying numbers of inputs and samples, but are primarily continuous input, binary classi cation problems. We i d e n tify three forms of bias that can a ect comparisons of this type (estimation, selection, and expert bias) and detail the methods used to avoid them. We compare and discuss the use of AUC with the conventional measure of classi er performance, overall accuracy (the probability of a correct response). It is found that AUC exhibits a number of desirable properties when compared to overall accuracy: increased sensitivity in Analysis of Variance (ANOVA) tests a standard error that decreased as both AUC and the number of test samples increased decision threshold independent invariant t o a priori class probabilities and it gives an indication of the amount o f \ w ork done" by a classi cation scheme, giving low scores to both random and \one class only" classi ers.It has been known for some time that AUC actually represents the probability that a randomly chosen positive example is correctly rated (ranked) with greater suspicion than a randomly chosen negative example. Moreover, this probability of correct ranking is the same quantity estimated by the non-parametric Wilcoxon statistic. We use this equivalence to show that the standard deviation of AUC, estimated using 10 fold cross validation, is a reliable estimator of the standard error estimated using the Wilcoxon test. The paper concludes with the recommendation that AUC be used in preference to overall accuracy when \single number" evaluation of machine learning algorithms is required. Draft Only 3 AbstractIn this paper we i n vestigate the use of the area under the receiver operating characteristic (ROC) curve ( A UC) as a performance measure for machine learning algorithms.As a case study we e v aluate six machine learning algorithms (C4.5, Multiscale Classi er, Perceptron, Multi-layer Perceptron, K-Nearest Neighbours, and a Quadratic Discriminant F unction) on six \real world" medical diagnostics data sets. We compare and discuss the use of AUC to the more conventional overall accuracy and nd that AUC exhibits a number of desirable properties when compared to overall accuracy: increased sensitivity in Analysis of Variance (ANOVA) tests a standard error that decreased as both AUC and the number of test samples increased decision threshold independent and it is invariant t o a priori class proba...

show abstract

Section: The Standard Error Of Aucmentioning

confidence: 99%

The use of the area under the ROC curve in the evaluation of machine learning algorithms

Bradley

1997

Pattern Recognition

5,593

2,981

View full text Add to dashboard Cite

show abstract

“…Under Gaussian assumptions, it is known that maximizing the deflection coefficient leads to the maximization of the detection performance in terms of the ROC curve [31]. In fact, it can be shown that for the centralized shift-in-mean Gaussian detection problem, the ROC curve in (19) could be expressed as [30] …”

Section: Measurement Of Detection Performancementioning

confidence: 99%

Cross-layer design for decentralized detection in WSNs

Tantawy¹,

Koutsoukos

Biswas

2014

EURASIP J. Adv. Signal Process.

View full text Add to dashboard Cite

A wireless sensor network (WSN) deployed for detection applications has the distinguishing feature that the sensors cooperate to perform the detection task. Therefore, the decoupled and maximum throughput design approaches typically used to design communication networks do not lead to the desired optimal detection performance. Recent work on decentralized detection has addressed the design of media access control (MAC) and routing protocols for detection applications by considering independently the quality of information (QoI), channel state information (CSI), and residual energy information (REI) for each sensor. However, little attention has been given to integrate the three quality measures (QoI, CSI, and REI) in the system design. In this work, we present a cross-layer approach to design a QoI, CSI, and REI-aware transmission control policy (XCP) that coordinates communication between local sensors and the fusion center, in order to maximize the detection performance. We formulate and solve a constrained non-linear optimization problem to find the optimal XCP design variables, for both ALOHA and time-division multiple access (TDMA) sensor networks. We show the detection performance gain compared to the typical decoupled and maximum throughput design approaches, without utilizing additional network resources. We compare ALOHA and TDMA MAC schemes and show the conditions under which each transmission scheme outperforms.

show abstract

“…as a performance criterion due to its simplicity and its strong relationship with the actual overall detection performance [13].…”

Section: B Sr Employed At the Fcmentioning

confidence: 99%

On noise-enhanced distributed inference in the presence of Byzantines

Gagrani

Sharma

Iyengar

et al. 2011

2011 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton)

View full text Add to dashboard Cite

Abstract-This paper considers the noise-enhanced distributed detection problem in the presence of Byzantine (malicious) nodes by suitably adding stochastic resonance (SR) noise. We consider two metrics -the minimum number of Byzantines (α blind ) needed to blind the fusion center as a security metric and the KullbackLeibler divergence (DKL) as a detection performance metric. We show that α blind increases when SR noise is added at the honest nodes. When Byzantines also start adding SR noise to their observations, we see no gain in terms of α blind . However, the detection performance of the network does improve with SR.We also consider a game theoretic formulation where this problem of distributed detection in the presence of Byzantines is modeled as a minimax game between the Byzantines and the inference network, and numerically find Nash equilibria.The case when SR noise is added to the signals received at the fusion center (FC) from the sensors is also considered. Our numerical results indicate that while there is no gain in terms of α blind , the network-wide performance measured in terms of the deflection coefficient does improve in this case.

show abstract

On deflection as a performance criterion in detection

Cited by 120 publications

References 6 publications

The use of the area under the ROC curve in the evaluation of machine learning algorithms

The use of the area under the ROC curve in the evaluation of machine learning algorithms

Cross-layer design for decentralized detection in WSNs

On noise-enhanced distributed inference in the presence of Byzantines

Contact Info

Product

Resources

About