On the relationship between effectiveness and accessibility

Azzopardi, Leif; Bache, Richard

doi:10.1145/1835449.1835667

Cited by 20 publications

(25 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, Azzopardi et al [2] showed that a positive relation exists between effectiveness and retrievability. Measuring effectiveness using precision at 10 (P@10) & Mean Average Precision (MAP), the results showed that as the effectiveness increases, the retrievability bias tends to decrease.…”

Section: Related Workmentioning

confidence: 99%

Quantifying retrieval bias in Web archive search

et al. 2017

View full text Add to dashboard Cite

A Web archive usually contains multiple versions of documents crawled from the Web at different points in time. One possible way for users to access a Web archive is through full-text search systems. However, previous studies have shown that these systems can induce a bias, known as the retrievability bias, on the accessibility of documents in community-collected collections (such as TREC collections). This bias can be measured by analyzing the distribution of the retrievability scores for each document in a collection, quantifying the likelihood of a document's retrieval. We investigate the suitability of retrievability scores in retrieval systems that consider every version of a document in a Web archive as an independent document. We show that the retrievability of documents can vary for different versions of the same document and that retrieval systems induce biases to different extents. We quantify this bias for a retrieval system which is adapted to handle multiple versions of the same document. The retrieval system indexes each version of a document independently, and we refine the search results using two techniques to aggregate similar versions. The first approach is to collapse similar versions of a document based on content similarity. The second approach is to collapse all versions of the same document based on their URLs. In both cases, we found that the degree of bias Universiteit Utrecht, Utrecht, The Netherlands is related to the aggregation level of versions of the same document. Finally, we study the effect of bias across time using the retrievability measure. Specifically, we investigate whether the number of documents crawled in a particular year correlates with the number of documents in the search results from that year. Assuming queries are not inherently temporal in nature, the analysis is based on the timestamps of documents in the search results returned using the retrieval model for all queries. The results show a relation between the number of documents per year and the number of documents retrieved by the retrieval system from that year. We further investigated the relation between the queries' timestamps and the documents' timestamps. First, we split the queries into different time frames using a 1-year granularity. Then, we issued the queries against the retrieval system. The results show that temporal queries indeed retrieve more documents from the assumed time frame. Thus, the documents from the same time frame were preferred by the retrieval system over documents from other time frames.

show abstract

Section: Related Workmentioning

confidence: 99%

Quantifying retrieval bias in Web archive search

et al. 2017

View full text Add to dashboard Cite

show abstract

“…Experiment 4: Altering the Number of Documents for Expansion The final experiment follows the same method as Experiment 3 however, the number of documents that terms are extracted from are altered. 10 terms are extracted from a varying number from the top x documents (1,2,3,4,5,6,7,8,9,10,12,15,20,25,50). …”

Section: Experiments 3: Altering the Number Of Terms For Expansionmentioning

confidence: 99%

“…where q is a query from the large query set Q. k dq is the rank at which d is retrieved given q, therefore the utility function f (k dq , c) determines the score that document d attains for query q given the rank cutoff c. r(d) is calculated by summing over all queries q in query set Q. Theoretically, Q represents the universe of all possible queries, but in practice Q is very large set of queries [1,2,4,5,10]. The standard measure of retrievability used employs the utility function f (k dq , c), such that if a document, d, is retrieved in the top c documents given q, then f (k dq , c) = 1, otherwise f (k dq , c) = 0.…”

Section: Introductionmentioning

confidence: 99%

An Initial Investigation of Query Expansion Bias

Wilkie

Azzopardi

2017

Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval

Self Cite

View full text Add to dashboard Cite

Query expansion is a useful retrieval mechanism for creating more verbose queries from the users initial key word search. Query expansion generally have multiple parameters that allow the user to define how many terms and where those terms come from are introduced to the expanded query. However, the idea that query expansion may be introducing biases into the system by selecting terms from overly retrievable documents has never been formally evaluated. In this work, the relationship between performance and retrievability bias is explored when various query expansion methods are employed to aide retrieval. Several parameters are altered, independently, to identify those that have an impact on bias. Parameters altered include; Rocchio's beta, length normalisation parameters, the number of terms added and the number of documents those terms are extracted from. The evaluation performed here identifies a strong correlation between performance and retrievability bias, suggesting that performance is increased by making the system more biased thus more likely to pick terms from a set of overly retrievable documents.

show abstract

“…Here we will describe and discuss the various efforts that have tried to understand the relationship between retrievability and effectiveness [19,18,40,4,39]. Firstly, from a theoretical point of view, we will discuss the different possible relationships and how retrievability can impact upon both effectiveness and efficiency.…”

Section: Relationships With Retrievability (1hr)mentioning

confidence: 99%

“…The conceptual / hypothesised relationship between retrievability and performance [4] 2. Retrievability and Efficiency 3.…”

Section: Relationships With Retrievability (1hr)mentioning

confidence: 99%

The retrievability of documents

Azzopardi

2014

Proceedings of the 37th International ACM SIGIR Conference on Research &Amp; Development in Information Retrieval

Self Cite

View full text Add to dashboard Cite

Retrievability is an important and interesting indicator that can be used in a number of ways to analyse Information Retrieval systems and document collections. Rather than focusing totally on relevance, retrievability examines what is retrieved, how often it is retrieved, and whether a user is likely to retrieve it or not. This is important because a document needs to be retrieved, before it can be judged for relevance. In this tutorial, we shall explain the concept of retrievability along with a number of retrievability measures, how it can be estimated and how it can be used for analysis. Since retrieval precedes relevance, we shall also provide an overview of how retrievability relates to effectivenessdescribing some of the insights that researchers have discovered thus far. We shall also show how retrievability relates to efficiency, and how the theory of retrievability can be used to improve both effectiveness and efficiency. Then we shall provide an overview of the different applications of retrievability such as Search Engine Bias, Corpus Profiling, etc., before wrapping up with challenges and opportunities. The final session of the day will look at example problems and ways to analyse and apply retrievability to other problems and domains. Participants are invited to bring their own problems to create a more focused practical session. This tutorial is ideal for: (i) researchers curious about retrievability and wanting to see how it can impact their research, (ii) researchers who would like to expand their set of analysis techniques, and/or (iii) researchers who would like to use retrievability to perform their own analysis.

show abstract

On the relationship between effectiveness and accessibility

Cited by 20 publications

References 5 publications

Quantifying retrieval bias in Web archive search

Quantifying retrieval bias in Web archive search

An Initial Investigation of Query Expansion Bias

The retrievability of documents

Contact Info

Product

Resources

About