Overview of the INEX 2008 Ad Hoc Track

Abstract

This paper gives an overview of the INEX 2008 Ad Hoc Track. The main goals of the Ad Hoc Track were twofold. The first goal was to investigate the value of the internal document structure (as provided by the XML markup) for retrieving relevant information. This is a continuation of INEX 2007 and, for this reason, the retrieval results are liberalized to arbitrary passages and measures were chosen to fairly compare systems retrieving elements, ranges of elements, and arbitrary passages. The second goal was to compare focused retrieval to article retrieval more directly than in earlier years. For this reason, standard document retrieval rankings have been derived from all runs, and evaluated with standard measures. In addition, a set of queries targeting Wikipedia have been derived from a proxy log, and the runs are also evaluated against the clicked Wikipedia pages. The INEX 2008 Ad Hoc Track featured three tasks: For the Focused Task a ranked-list of nonoverlapping results (elements or passages) was needed. For the Relevant in Context Task non-overlapping results (elements or passages) were returned grouped by the article from which they came. For the Best in Context Task a single starting point (element start tag or passage start) for each article was needed. We discuss the results for the three tasks, and examine the relative effectiveness of element and passage retrieval. This is examined in the context of content only (CO, or Keyword) search as well as content and structure (CAS, or structured) search. Finally, we look at the ability of focused retrieval techniques to rank articles, using standard document retrieval techniques, both against the judged topics as well as against queries and clicks from a proxy log.

XML Retrieval Using Pruned Element-Index Files

“…That is, focused retrieval based on I full yields an iP figure of 0.643 at 1% recall level. The best official result in INEX 2008 for this task is 0.689 and our result is within the top-10 results of this task (see Table 6 in [15]). This is also the case for RiC and BiC results that will be discussed in the upcoming sections, proving that we have a reasonable baseline for drawing conclusions in our experimental framework.…”

“…In this study, we concentrate on three ad-hoc retrieval tasks, namely, Focused, Relevant-in-Context (RiC) and Best-in-Context (BiC), as described in recent INEX campaigns (e.g., see [13,15]). In short, the Focused task is designed to retrieve the most focused results for a query without returning overlapping elements.…”

“…However, while ranking the documents, if the article node of the document is within these retrieved elements, we use its score as the document score. Otherwise, we use the score of the highest scoring retrieved element as the score of [15]. Note that, this suggests starting to read each ranked document from the beginning.…”

“…Note that, this suggests starting to read each ranked document from the beginning. For our work, we also experimented with providing the offset of the highest scoring element per document as BEP [15], which yielded inferior results to the former approach. Thus, we only report the results where BEP is set to 1.…”

DCU and ISI@INEX 2010: Adhoc and Data-Centric Tracks

“…Since INEX 2007 [1] arbitrary passages are also permitted as retrievable units, besides XML elements. A retrieved passage can be a sequence of textual content either from within an element or spanning a range of elements.…”