Threshold Average Precision (TAP-k): a measure of retrieval designed for bioinformatics

Abstract

Motivation: Since database retrieval is a fundamental operation, the measurement of retrieval efficacy is critical to progress in bioinformatics. This article points out some issues with current methods of measuring retrieval efficacy and suggests some improvements. In particular, many studies have used the pooled receiver operating characteristic for n irrelevant records (ROCn) score, the area under the ROC curve (AUC) of a ‘pooled’ ROC curve, truncated at n irrelevant records. Unfortunately, the pooled ROCn score does not faithfully reflect actual usage of retrieval algorithms. Additionally, a pooled ROCn score can be very sensitive to retrieval results from as little as a single query.Methods: To replace the pooled ROCn score, we propose the Threshold Average Precision (TAP-k), a measure closely related to the well-known average precision in information retrieval, but reflecting the usage of E-values in bioinformatics. Furthermore, in addition to conditions previously given in the literature, we introduce three new criteria that an ideal measure of retrieval efficacy should satisfy.Results: PSI-BLAST, GLOBAL, HMMER and RPS-BLAST provided examples of using the TAP-k and pooled ROCn scores to evaluate sequence retrieval algorithms. In particular, compelling examples using real data highlight the drawbacks of the pooled ROCn score, showing that it can produce evaluations skewing far from intuitive expectations. In contrast, the TAP-k satisfies most of the criteria desired in an ideal measure of retrieval efficacy.Availability and Implementation: The TAP-k web server and downloadable Perl script are freely available at http://www.ncbi.nlm.nih.gov/CBBresearch/Spouge/html.ncbi/tap/Contact: spouge@ncbi.nlm.nih.govSupplementary Information: Supplementary data are available at Bioinformatics online.

Improving Retrieval Efficacy of Homology Searches Using the False Discovery Rate

“…In this study, we utilize the Threshold Average Precision (TAP) [14] method as the evaluation criterion for retrieval efficacy. The TAP method calculates the median Average Precision-Recall with a moderate adjustment for irrelevant sequences just before the threshold.…”

Time-Aware Ranking in Dynamic Citation Networks

“…Usually, the area under the receiver operating characteristic curve (AUC) score is the most popular criterion for the task. However, it was shown that AUC may fail to faithfully reflect the actual quality when the AUC scores are pooled together to evaluate a retrieval system for multiple independent retrieval tasks [16]. AUC is not robust against outlier results.…”

“…Finally, AUC does not always decrease as the threshold relaxed to include the entire retrieval list. To address these issues, a new evaluation method called the threshold average precision (TAP-k) was proposed [16]. We will adopt this new method to evaluate the metrics on their performance for predicting top 10% of highly cited articles.…”

“…TAP-k will penalize ranked lists that are cut short prematurely in an attempt to boost its precision and ranked lists with scores that only reflect the rank but not the quality or importance of the retrieved items (in our case, the papers). See [16] for details. Again, we applied each metric to rank the oldest 90% of the papers.…”

Ranking Interactions for a Curation Task

“…The implicit assumption is that a curator could use the ranking to decide where to stop looking at the results, therefore a better ranking provides a better user experience. A recently proposed alternative measure of the ranking of the results is the "Threshold Average Precision" (TAP-k) [9], which (in slightly simplified terms) averages precision for the results above a given error threshold. The TAP-k metric is easier to interpret and directly relevant for the end user, who in most cases would not be willing to inspect a long list of results containing many false positives.…”