Overview of the INEX 2010 Book Track: Scaling Up the Evaluation Using Crowdsourcing

Abstract

Abstract. The goal of the INEX Book Track is to evaluate approaches for supporting users in searching, navigating and reading the full texts of digitized books. The investigation is focused around four tasks: 1) Best Books to Reference, 2) Prove It, 3) Structure Extraction, and 4) Active Reading. In this paper, we report on the setup and the results of these tasks in 2010. The main outcome of the track lies in the changes to the methodology for constructing the test collection for the evaluation of the Best Books and Prove It search tasks. In an effort to scale up the evaluation, we explored the use of crowdsourcing both to create the test topics and then to gather the relevance labels for the topics over a corpus of 50k digitized books. The resulting test collection construction methodology combines editorial judgments contributed by INEX participants with crowdsourced relevance labels. We provide an analysis of the crowdsourced data and conclude that -with appropriate task designcrowdsourcing does provide a suitable framework for the evaluation of book search approaches.

Report on INEX 2010

“…In this section, we will briefly discuss the BB and PI tasks. Further details on all four tasks are available in [8].…”

Searching online book documents and analyzing book citations

“…Querying using the 4905 ISBNs other well known sources, including Abebooks 15 , Amazon book 16 , ISBNSearch 17 , and BookFinder4U 18 , we retrieved 814, 1275, 1170, 1301 books respectively while Google Books returned 4329 books with valid title and authors and which covered all others. The first four pages of these 4329 books are then extracted line by line and represented by features shown in Table 1.…”

“…However, the evaluation of search and retrieval over large book repositories is still a difficult task. G. Kazai et al used crowdsourcing for book search evaluation and found that well designed crowdsourcing can be an effective tool for the evaluation of book IR systems [15,16]. A similar problem has also been tested in the social book search task [17].…”

Learning to rank by aggregating expert preferences

“…For crowdsourcing experiments we use the dataset from Task 2 of the TREC2011 Crowdsourcing Track 1 [17]. The dataset is a collection of binary (relevant or not relevant) judgements from 762 workers for 19,033 documents.…”

“…This transformation makes it possible to apply any learningto-rank method to optimize the parameters of the aggregating function for the target metric. Experiments on crowdsourcing task from TREC2011 [17] and meta-search tasks with Microsoft's LETOR4.0 [20] data sets show that our model significantly outperforms existing aggregation methods.…”