Classifier Loss Under Metric Uncertainty

Skalak, David B.; Niculescu-Mizil, Alexandru; Caruana, Rich

doi:10.1007/978-3-540-74958-5_30

Cited by 5 publications

(8 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several studies have found that optimizing a metric different from the metric being evaluated can bring better results than optimizing the same metric [15,27,29,33]. On the contrary, a recent study claim that such findings come from the use of wrong statistical testing strategy [16].…”

Section: Related Workmentioning

confidence: 99%

AMORI: A Metric-based One Rule Inducer

Lavesson¹,

Davidsson²

2009

Proceedings of the 2009 SIAM International Conference on Data Mining

View full text Add to dashboard Cite

The requirements of real-world data mining problems vary extensively. It is plausible to assume that some of these requirements can be expressed as application-specific performance metrics. An algorithm that is designed to maximize performance given a certain learning metric may not produce the best possible result according to these applicationspecific metrics. We have implemented A Metric-based One Rule Inducer (AMORI), for which it is possible to select the learning metric. We have compared the performance of this algorithm by embedding three different learning metrics (classification accuracy, the F-measure, and the area under the ROC curve), on 19 UCI data sets. In addition, we have compared the results of AMORI with those obtained using an existing rule learning algorithm of similar complexity (One Rule) and a state-of-the-art rule learner (Ripper). The experiments show that a performance gain is achieved, for all included metrics, when using identical metrics for learning and evaluation. We also show that each AMORI/metric combination outperforms One Rule when using identical learning and evaluation metrics. The performance of AMORI is acceptable when compared with Ripper. Overall, the results suggest that metric-based learning is a viable approach.

show abstract

Section: Related Workmentioning

confidence: 99%

AMORI: A Metric-based One Rule Inducer

Lavesson¹,

Davidsson²

2009

Proceedings of the 2009 SIAM International Conference on Data Mining

View full text Add to dashboard Cite

show abstract

“…The results are quite surprising and puzzling, and no convincing explanations were given. Nevertheless, several other papers [1,3,4] have since confirmed his finding. For example, Huang and Ling [3] studied model selection with many popular machine learning metrics and claimed that often an evaluation metric different from the goal metric can better select the correct models.…”

Section: Introductionmentioning

confidence: 64%

“…For example, Huang and Ling [3] studied model selection with many popular machine learning metrics and claimed that often an evaluation metric different from the goal metric can better select the correct models. Skalak and Caruana [1] used absolute loss to compare model selection abilities of various metrics, and drew similar conclusions. It now seems to be a well-regarded conclusion in the machine learning community that a different metric can do a better job in model selection.…”

Section: Introductionmentioning

confidence: 74%

“…Skalak and Caruana [1] studied the robustness of the evaluation metric when the goal metric is unknown. They used the absolute loss for measuring the model selection error but again no variance is calculated for the loss.…”

Section: Review Of Previous Workmentioning

confidence: 99%

“…For example, if the goal is to obtain a classification model with 1 Other empirical approaches are reviewed in Section 2. The goal metric is also called performance metric, and the evaluation metric is called selection metric [1].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Proper Model Selection with Significance Test

Huang

Ling

Zhang

et al.

Machine Learning and Knowledge Discovery in Databases

View full text Add to dashboard Cite

Abstract. Model selection is an important and ubiquitous task in machine learning. To select models with the best future classification performance measured by a goal metric, an evaluation metric is often used to select the best classification model among the competing ones. A common practice is to use the same goal and evaluation metric. However, in several recent studies, it is claimed that using an evaluation metric (such as AUC) other than the goal metric (such as accuracy) results in better selection of the correct models. In this paper, we point out a flaw in the experimental design of those studies, and propose an improved method to test the claim. Our extensive experiments show convincingly that only the goal metric itself can most reliably select the correct classification models.

show abstract

Loss functions for predicted click‐through rates in auctions for online advertising

Hummel

McAfee

2017

J of Applied Econometrics

View full text Add to dashboard Cite

Abstract. We characterize the optimal loss functions for predicted click-through rates in auctions for online advertising. While standard loss functions such as mean squared error or log likelihood severely penalize large mispredictions while imposing little penalty on smaller mistakes, a loss function reflecting the true economic loss from mispredictions imposes significant penalties for small mispredictions and only slightly larger penalties on large mispredictions. We illustrate that when the model is misspecified, using such a loss function can improve economic efficiency, but the efficiency gain is likely to be small.

show abstract

Classifier Loss Under Metric Uncertainty

Cited by 5 publications

References 8 publications

AMORI: A Metric-based One Rule Inducer

AMORI: A Metric-based One Rule Inducer

Proper Model Selection with Significance Test

Loss functions for predicted click‐through rates in auctions for online advertising

Contact Info

Product

Resources

About