A General Theory of IR Evaluation Measures

Ferrante, Marco; Ferro, Nicola; Pontarollo, Silvia

doi:10.1109/tkde.2018.2840708

Cited by 24 publications

(44 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…by using correlation analysis [119], discriminative power [86,87], or robustness to pool downsampling [16,124]. On the other hand, few studies have been undertaken to understand the formal properties of evaluation measures and they have just scratched the surface of the problem: [13,19,7,110,39,40,41].…”

Section: Experimental Evaluation In Irmentioning

confidence: 99%

“…Scales have increasing properties: a nominal scale allows for determination of equality and for the computation of the mode; an ordinal scale allows only for determination of greater or less and for the computation of medians and percentiles; an interval scale allows also for determination of equality of intervals or differences and for the computation of mean, standard deviation, and rank-order correlation; finally, a ratio scale allows also for the determination of equality of ratios and for the computation of coefficient of variation. Recently, Ferrante et al [40,41] have theoretically shown that some of the most known and used IR measures, like Average Precision (AP) or Discounted Cumulative Gain (DCG), are not interval-scales. As a consequence, according to Stevens's prescriptions, we should neither compute means, standard deviations and confidence intervals, nor perform significance tests that require an interval scale.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Towards Meaningful Statements in IR Evaluation: Mapping Evaluation Measures to Interval Scales

Ferrante

Ferro²,

Fuhr

2021

IEEE Access

Self Cite

View full text Add to dashboard Cite

Information Retrieval (IR) is a discipline deeply rooted in evaluation since its inception. Indeed, experimentally measuring and statistically validating the performance of IR systems are the only possible ways to compare systems and understand which are better than others and, ultimately, more effective and useful for end-users. Since the seminal paper by Stevens [103], it is known that the properties of the measurement scales determine the operations you should or should not perform with values from those scales. For example, Stevens suggested that you can compute means and variances only when you are working with, at least, interval scales. It was recently shown that the most popular evaluation measures in IR are not intervalscaled. However, so far, there has been little or no investigation in IR on the impact and consequences of departing from scale assumptions. Taken to the extremes, it might even mean that decades of experimental IR research used potentially improper methods, which may have produced results needing further validation. However, it was unclear if and to what extent these findings apply to actual evaluations; this opened a debate in the community with researchers standing on opposite positions about whether this should be considered an issue (or not) and to what extent. In this paper, we first give an introduction to the representational measurement theory explaining why certain operations and significance tests are permissible only with scales of a certain level. For that, we introduce the notion of meaningfulness specifying the conditions under which the truth (or falsity) of a statement is invariant under permissible transformations of a scale. Furthermore, we show how the recall base and the length of the run may make comparison and aggregation across topics problematic. Then we propose a straightforward and powerful approach for turning an evaluation measure into an interval scale, and describe an experimental evaluation of the differences between the original measures and the intervalscaled ones. For all the regarded measures -namely Precision, Recall, Average Precision, (Normalized) Discounted Cumulative Gain, Rank-Biased Precision and Reciprocal Rank -we observe substantial effects, both on the order of average values and on the outcome of significance tests. For the latter, previously significant differences turn out to be insignificant, while insignificant ones become significant. The effect varies remarkably between the tests considered but on average, we observed a 25% change in the decision about which systems are significantly different and which are not. These experimental findings further support the idea that measurement scales matter and that departing from their assumptions has an impact. This not only suggests that, to the extent possible, it would be better to comply with such assumptions but it also urges us to clearly indicate when we depart from such assumptions and, carefully, point out the limitations of the conclusions we draw and under which conditions they are drawn.

show abstract

Section: Experimental Evaluation In Irmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Towards Meaningful Statements in IR Evaluation: Mapping Evaluation Measures to Interval Scales

Ferrante

Ferro²,

Fuhr

2021

IEEE Access

Self Cite

View full text Add to dashboard Cite

show abstract

“…This paper follows in the tradition of the so-called "axiomatic" approach to "evaluating evaluation" in information retrieval (see e.g., (Amigó et al 2011;Busin and Mizzaro 2013;Ferrante et al 2015Ferrante et al , 2018Moffat 2013;Sebastiani 2015)), which is based on describing (and often: arguing in favour of) a number of properties (that most of this literature calls -perhaps improperly -"axioms") that an evaluation measure for the task being considered should intuitively satisfy. The benefit of this approach is that it shifts the discussion from the evaluation measures to their properties, which amounts to shifting the discussion from a complex construction to its building blocks: once the scientific community has agreed on a set of properties (the building blocks), it then follows whether a given measure (the construction) is satisfactory or not.…”

Section: Introductionmentioning

confidence: 99%

Evaluation measures for quantification: an axiomatic approach

Sebastiani

2019

Inf Retrieval J

View full text Add to dashboard Cite

Quantification is the task of estimating, given a set σ of unlabelled items and a set of classes C = {c 1 , . . . , c |C| }, the prevalence (or "relative frequency") in σ of each class c i ∈ C. While quantification may in principle be solved by classifying each item in σ and counting how many such items have been labelled with c i , it has long been shown that this "classify and count" (CC) method yields suboptimal quantification accuracy. As a result, quantification is no longer considered a mere byproduct of classification, and has evolved as a task of its own. While the scientific community has devoted a lot of attention to devising more accurate quantification methods, it has not devoted much to discussing what properties an evaluation measure for quantification (EMQ) should enjoy, and which EMQs should be adopted as a result. This paper lies down a number of interesting properties that an EMQ may or may not enjoy, discusses if (and when) each of these properties is desirable, surveys the EMQs that have been used so far, and discusses whether they enjoy or not the above properties. As a result of this investigation, some of the EMQs that have been used in the literature turn out to be severely unfit, while others emerge as closer to what the quantification community actually needs. However, a significant result is that no existing EMQ satisfies all the properties identified as desirable, thus indicating that more research is needed in order to identify (or synthesize) a truly adequate EMQ.

show abstract

“…Evaluation measures are an intrinsic part of experimental evaluation. Even if a growing attention is called in the field for developing stronger theoretical foundations [1][2][3]9,12], they are often formulated and justified in a somewhat informal and intuitive way rather then being based on well-founded mathematical models. Carterette [4] has made a post-hoc attempt to propose a unifying framework which explains modern evaluation measures based on three components: a browsing model, a model of document utility, and a utility accumulation model.…”

Section: Introductionmentioning

confidence: 99%

A Markovian Approach to Evaluate Session-Based IR Systems

Dijk

Ferrante

Ferro

et al. 2019

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

We investigate a new approach for evaluating session-based information retrieval systems, based on Markov chains. In particular, we develop a new family of evaluation measures, inspired by random walks, which account for the probability of moving to the next and previous documents in a result list, to the next query in a session, and to the end of the session. We leverage this Markov chain to substitute what in existing measures is a fixed discount linked to the rank of a document or to the position of a query in a session with a stochastic average time to reach a document and the probability of actually reaching a given query. We experimentally compare our new family of measures with existing measures -namely, session DCG, Cube Test, and Expected Utilityover the TREC Dynamic Domain track, showing the flexibility of the proposed measures and the transparency in modeling the user dynamics.

show abstract

A General Theory of IR Evaluation Measures

Cited by 24 publications

References 21 publications

Towards Meaningful Statements in IR Evaluation: Mapping Evaluation Measures to Interval Scales

Towards Meaningful Statements in IR Evaluation: Mapping Evaluation Measures to Interval Scales

Evaluation measures for quantification: an axiomatic approach

A Markovian Approach to Evaluate Session-Based IR Systems

Contact Info

Product

Resources

About