Overview of INEX 2014

Abstract

Abstract. INEX investigates focused retrieval from structured documents by providing large test collections of structured documents, uniform evaluation measures, and a forum for organizations to compare their results. This paper reports on the INEX 2014 evaluation campaign, which consisted of three tracks: The Interactive Social Book Search Track investigated user information seeking behavior when interacting with various sources of information, for realistic task scenarios, and how the user interface impacts search and the search experience. The Social Book Search Track investigated the relative value of authoritative metadata and usergenerated content for search and recommendation using a test collection with data from Amazon and LibraryThing, and user profiles and personal catalogues. The Tweet Contextualization Track investigated tweet contextualization, helping a user to understand a tweet by providing him with a short background summary generated from relevant Wikipedia passages aggregated into a coherent summary. INEX 2014 was an exciting year for INEX in which we for the third time ran our workshop as part of the CLEF labs in order to facilitate knowledge transfer between the evaluation forums. This paper gives an overview of all the INEX 2014 tracks, their aims and task, the built test-collections, the participants, and gives an initial analysis of the results.

Show Me Your Evidence - an Automatic Method for Context Dependent Evidence Detection

“…Probably the closest of which is the relatively unexplored task of Evidence Retrieval (ER) (Cartright et al, 2011;Bellot et al, 2013). However, while ER focus is on identifying whole documents, in CDED the goal is to pinpoint a typically much shorter text segment which can be used directly to support a claim.…”

A Method for Short Message Contextualization: Experiments at CLEF/INEX

“…In 2013 the informativeness was estimated as the overlap of a summary with 3 pools of relevant passages: (1) prior set (PRIOR) of relevant pages selected by organizers (40 tweets, 380 passages); (2) pool selection (POOL) of the most relevant passages (1 760) from participant submissions for 45 selected tweets; and (3) all relevant texts (ALL) merged together with extra passages from a random pool of 10 tweets (70 tweets, 2 378 relevant passages) [2]. The system was evaluated with three parameter sets.…”

“…Informativeness was estimated as the lexical overlap (uni, big and skip representing the proportion of shared unigrams, bigrams and bigrams with gaps of two tokens respectively) of a summary with the pool of relevant passages extracted from the runs submitted by all participants [2]. Official ranking was based on decreasing score of divergence with the gold standard estimated by skip:…”

“…Meij et al mapped a tweet into a set of Wikipedia articles but in their work, no summary is provided to the user, rather a set of related links [6]. San Juan et al went a step further and introduced Tweet Contextualization (TC) as an INEX task which became the CLEF lab in 2012 [9,2].…”

The Role of Language Skills in Interactive Social Book Search Sessions

“…The overall goal of the Social Book Search lab at CLEF (Conference and Labs of the Evaluation Forum) in 2014 [1] and 2015 [6] was to investigate how professional metadata (title, authors, ...) can be combined with social meta-data (tags, reviews) to satisfy an information need. Within this the Interactive Social Book Search Task (iSBS) looked at how the two types of meta-data can be combined in the search interface and in what way users make use of the two meta-data searches when interacting with the interface to complete a task.…”