Adjusting the Outputs of a Classifier to New a Priori Probabilities: A Simple Procedure

Abstract

It sometimes happens (for instance in case control studies) that a classifier is trained on a data set that does not reflect the true a priori probabilities of the target classes on real-world data. This may have a negative effect on the classification accuracy obtained on the real-world data set, especially when the classifier's decisions are based on the a posteriori probabilities of class membership. Indeed, in this case, the trained classifier provides estimates of the a posteriori probabilities that are not valid for this real-world data set (they rely on the a priori probabilities of the training set). Applying the classifier as is (without correcting its outputs with respect to these new conditions) on this new data set may thus be suboptimal. In this note, we present a simple iterative procedure for adjusting the outputs of the trained classifier with respect to these new a priori probabilities without having to refit the model, even when these probabilities are not known in advance. As a by-product, estimates of the new a priori probabilities are also obtained. This iterative algorithm is a straightforward instance of the expectation-maximization (EM) algorithm and is shown to maximize the likelihood of the new data. Thereafter, we discuss a statistical test that can be applied to decide if the a priori class probabilities have changed from the training set to the real-world data. The procedure is illustrated on different classification problems involving a multilayer neural network, and comparisons with a standard procedure for a priori probability estimation are provided. Our original method, based on the EM algorithm, is shown to be superior to the standard one for a priori probability estimation. Experimental results also indicate that the classifier with adjusted outputs always performs better than the original one in terms of classification accuracy, when the a priori probability conditions differ from the training set to the real-world data. The gain in classification accuracy can be significant.

A Review on Quantification Learning

“…The main reason is that those methods were not tested as quantifiers. This is for instance the case of [Chan and Ng 2006] in which [Vucetic and Obradovic 2001;Saerens et al 2002] were applied to boost the accuracy of word sense disambiguation systems.…”

“…The goal of [AlaizRodríguez et al 2008] is to design an automatic process to quantify cells proportions. The authors use a combination of computer vision techniques and quantification algorithms [Saerens et al 2002;Forman 2005], see Sections 6.2 and 8.1. The experimental results reported with boar sperm samples using such techniques outperform previous approaches based on classification in terms of several measures, including mean absolute error, KL divergence and mean relative error.…”

“…The study deals with two large datasets: a survey of coral reefs from Caribbean sea, and a time-series dataset of plankton samples. The authors investigate several quantification methods from literature showing, for instance, that the EM algorithm proposed in [Saerens et al 2002] outperform other approaches for the plankton dataset. For the same type of applications, propose a methodology to assess the efficacy of quantifiers which takes into account the fact that the data distribution may vary between the training phase and once the model is deployed.…”

“…For instance, quantifying consumers' opinion belongs to X → Y problems, while quantifying plankton morphological groups belongs to Y → X problems. However, one of the main reason to identify quantification tasks with Y → X problems is because almost all quantification algorithms [Saerens et al 2002;Forman 2008;González-Castro et al 2013] assume that P (x|y) does not change. This learning assumption seems easier to be fulfilled in Y → X problems because y determines x.…”

Uncertainty-aware estimation of population abundance using machine learning

“…In [26], bias correction is based on the estimated decisions of the classifier. While in [23], authors estimate the a priori distribution of a new dataset based on the features. The main difference with previous approaches is that we use the similarity scores from automatic classification methods to determine the counts, while previous methods work either directly at the decision level (having less information) or at the feature level (can not use classifier output).…”