Information retrieval evaluation using test collections

Scholer, Falk; Kelly, Diane; Carterette, Ben

doi:10.1007/s10791-016-9281-7

Cited by 15 publications

(6 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…. $15.00 https://doi.org/10.1145/3209978.3210024 then, many evaluation metrics have been proposed to measure the effectiveness of information retrieval systems [20,22,27].…”

Section: Introductionmentioning

confidence: 99%

An Axiomatic Analysis of Diversity Evaluation Metrics

Amigó

Spina

Carrillo-de-Albornoz

2018

The 41st International ACM SIGIR Conference on Research &Amp; Development in Information Retrieval

View full text Add to dashboard Cite

Many evaluation metrics have been defined to evaluate the effectiveness ad-hoc retrieval and search result diversification systems. However, it is often unclear which evaluation metric should be used to analyze the performance of retrieval systems given a specific task. Axiomatic analysis is an informative mechanism to understand the fundamentals of metrics and their suitability for particular scenarios. In this paper, we define a constraint-based axiomatic framework to study the suitability of existing metrics in search result diversification scenarios. The analysis informed the definition of Rank-Biased Utility (RBU) -an adaptation of the wellknown Rank-Biased Precision metric -that takes into account redundancy and the user effort associated to the inspection of documents in the ranking. Our experiments over standard diversity evaluation campaigns show that the proposed metric captures quality criteria reflected by different metrics, being suitable in the absence of knowledge about particular features of the scenario under study.

show abstract

“…. $15.00 https://doi.org/10.1145/3209978.3210024 then, many evaluation metrics have been proposed to measure the effectiveness of information retrieval systems [20,22,27].…”

Section: Introductionmentioning

confidence: 99%

An Axiomatic Analysis of Diversity Evaluation Metrics

Amigó

Spina

Carrillo-de-Albornoz

2018

The 41st International ACM SIGIR Conference on Research &Amp; Development in Information Retrieval

View full text Add to dashboard Cite

show abstract

“…Information retrieval technology has experienced the development of early information retrieval 21,22 technology to computer modern retrieval technology. Before the computer retrieval technology was produced, the information retrieval technology generally experienced the development stage from the complete manual retrieval system → semi-mechanical retrieval system → electromechanical and photoelectric retrieval systems.…”

Section: Information Retrieval Technologymentioning

confidence: 99%

Probability model of sensitive similarity measures in information retrieval

Zhang

2020

International Journal of Advanced Robotic Systems

View full text Add to dashboard Cite

In today’s Internet age, a lot of data is stored and used, which is very important. In people’s daily life, if these data are sorted, information retrieval technology will be used, and in information retrieval, some information retrieval inaccuracies often appear. Information retrieval model is an important framework and method for fast, complete, and accurate user information retrieval. With the rapid development of information technology, great changes have taken place in people’s production and life. Various information network technologies are widely used in people’s lives. The resulting flow of information shows explosive growth, information retrieval. User requirements are getting higher and higher. How to complete personalized information retrieval in a large amount of mixed information, so that retrieval technology can help us obtain effective retrieval results, has become a realistic problem worth exploring. In this article, the application of probability model based on sensitive similarity measure in information retrieval model is analyzed, and a similarity measure algorithm based on spectral clustering is proposed. By improving the similarity measure, the sensitivity problem of scale parameters is overcome and the retrieval precision is improved. In order to better reflect the superiority of the proposed algorithm, this article compares with ng-jordan-weiss (NJW) and deep sparse subspace clustering (DSSC) algorithms. The experimental results show that the proposed algorithm is superior to NJW and DSSC algorithms for different data sets in different evaluation indicators (Rand and F-measure).

show abstract

“…Lack of comparable results in information retrieval have been observed and investigated in the IR community. A brief list of evaluation issues using TREC like test collections can be found here [28]. We are not focusing the test collections creation issue in this work.…”

Section: Evaluation In Information Retrievalmentioning

confidence: 99%

Benchmarking Clinical Decision Support Search

Nguyen¹,

Karimi²,

Falamaki³

et al. 2018

Preprint

View full text Add to dashboard Cite

Finding relevant literature underpins the practice of evidence-based medicine. From 2014 to 2016, TREC conducted a clinical decision support track, wherein participants were tasked with finding articles relevant to clinical questions posed by physicians. In total, 87 teams have participated over the past three years, generating 395 runs. During this period, each team has trialled a variety of methods. While there was significant overlap in the methods employed by different teams, the results were varied. Due to the diversity of the platforms used, the results arising from the different techniques are not directly comparable, reducing the ability to build on previous work. By using a stable platform, we have been able to compare different document and query processing techniques, allowing us to experiment with different search parameters. We have used our system to reproduce leading team's runs, and compare the results obtained. By benchmarking our indexing and search techniques, we can statistically test a variety of hypotheses, paving the way for further research.

show abstract

Information retrieval evaluation using test collections

Cited by 15 publications

References 7 publications

An Axiomatic Analysis of Diversity Evaluation Metrics

An Axiomatic Analysis of Diversity Evaluation Metrics

Probability model of sensitive similarity measures in information retrieval

Benchmarking Clinical Decision Support Search

Contact Info

Product

Resources

About