The Importance of the Test Set Size in Quantification Assessment

Abstract

<jats:p>Quantification is a task similar to classification in the sense that it learns from a labeled training set. However, quantification is not interested in predicting the class of each observation, but rather measure the class distribution in the test set. The community has developed performance measures and experimental setups tailored to quantification tasks. Nonetheless, we argue that a critical variable, the size of the test sets, remains ignored. Such disregard has three main detrimental effects. First, it implicitly assumes that quantifiers will perform equally well for different test set sizes. Second, it increases the risk of cherry-picking by selecting a test set size for which a particular proposal performs best. Finally, it disregards the importance of designing methods that are suitable for different test set sizes. We discuss these issues with the support of one of the broadest experimental evaluations ever performed, with three main outcomes. (i) We empirically demonstrate the importance of the test set size to assess quantifiers. (ii) We show that current quantifiers generally have a mediocre performance on the smallest test sets. (iii) We propose a metalearning scheme to select the best quantifier based on the test size that can outperform the best single quantification method.</jats:p>

Minimising quantifier variance under prior probability shift

“…Further research on developing efficient quantifiers for small or moderate training and test data set sizes therefore is highly desirable. A promising step in this direction has already been done by [13], with a proposal for the selection of the most suitable quantifiers for problems on data sets with widely varying sizes.…”

An efficient and accurate method for binary quantification

“…Several researchers have investigated how the experimental design decisions may influence the performance of quantifiers. A recent example is the analysis of the influence of test set size (MALETZKE et al, 2020) on the quantification performance.…”

“…The most prominent is that the classification task processes a single instance to get the final output, while the quantification task processes a group of instances. The group can be of different sizes, and the performance of quantification methods varies with the change in size (MALETZKE et al, 2020). In addition, large volume of data and data stream applications demand methods that can process the data and estimate the output efficiently and accurately.…”

“…Therefore, we should consider such differences when evaluating quantification methods. In particular, the size of the test set is a data characteristic that can affect the performance of quantifiers (MALETZKE et al, 2020). However, that parameter is usually not a point of concern for classification models because classifiers give predictions for individual instances.…”

“…Thus, the size of the test set influences the quality of a quantification model. Therefore, it must be necessary to consider test sets sizes to evaluate quantification methods (MALETZKE et al, 2020).…”