A Component-Level Analysis of an Academic Search Test Collection.

Abstract

This study analyzes causes for successful and unsuccessful search performance in an academic search test collection. Based on a component-level evaluation setting presented in a parallel paper, analyses of the recall base and the semantic heterogeneity of queries and documents were used for performance prediction. The study finds that neither the recall base, query specificity nor ambiguity can predict the overall search performance or identify badly performing queries. A detailed query analysis finds patterns for negative effects (e.g. non-contentbearing terms in topics), but none are overly significant.

A Component-Level Analysis of an Academic Search Test Collection.

“…While component-level evaluation as presented in another paper [4] focuses on the IR system components, but neglects to delve more deeply into query or document analyses, this paper delves deeper into the query and document analysis.…”

“…Some documents contain more content than others, making them more likely to be retrieved. The test collection and component-level experimental setup is described in more detail in [4].…”

“…For the experiments, the open source retrieval kit Lemur 1 was used. For the component-level evaluation [4], nearly every possible option of Lemur in combination with the document and topic fields was used for retrieval, resulting in a total of around 327,600 data points. The retrieval experiments conducted with Lemur were evaluated with the trec eval program 2 .…”

“…Academic search is defined as the domain of IR, which concerns itself with searching scientific data, mostly research output in the form of publications [12]. Both information needs and document collections present unique challenges in academic search: document collections are smaller than web search or newspaper collections; document and query terminology can be highly technical and domain-specific; documents may contain only sparse text (just metadata); and information needs can be very specific with subsequent fewer relevant results in smaller collections [4].…”

“…In a parallel paper [4], we present the system configurations that were tested in a component-level setting, while in this paper, query performance indicators for an academic search collection are studied.…”