On the role of user-centred evaluation in the advancement of interactive information retrieval

Abstract

This paper discusses the role of user-centred evaluations as an essential method for researching interactive information retrieval. It draws mainly on the work carried out during the Clarity Project where different user-centred evaluations were used during the lifecycle of a cross-language information retrieval system. The iterative testing was not only instrumental to the development of a usable system, but it enhanced our knowledge of the potential, impact, and actual use of cross-language information retrieval technology. Indeed the role of the user evaluation was dual: by testing a specific prototype it was possible to gain a micro-view and assess the effectiveness of each component of the complex system; by cumulating the result of all the evaluations (in total 43 people were involved) it was possible to build a macro-view of how cross-language retrieval would impact on users and their tasks. By showing the richness of results that can be acquired, this paper aims at stimulating researchers into considering user-centred evaluations as a flexible, adaptable and comprehensive technique for investigating non-traditional information access systems.

A social approach to context-aware retrieval

“…As first step we want to exploit benchmarks to evaluate detailed implementation solutions, like, for example, different algorithms to assess the relevance of tags for situations and resources. After that, we plan to apply an IIR evaluation methodology, involving users in a controlled environments, following the ideas presented [1,10]. Finally a broader user-centred evaluation will help us to understand if the sCAB is effective in the real world.…”

An overview of semantic search evaluation initiatives

“…Evaluation from a user-oriented perspective is important in going beyond retrieval effectiveness to assess retrieval performance, for example assessing user satisfaction with the results, usability of the interface, whether users are engaged with the system, user performance with a task and the effects of changes in the retrieval system on user behaviour (Tague and Schultz, 1989;Saracevic, 1995;Harter and Hert, 1997;Su, 1992;Saracevic, 1995;Voorhees, 2002;Ingwersen and Järvelin, 2005). This requires going beyond the traditional Cranfield-style IR experiment and various studies have been carried out from an Interactive Information Retrieval (IIR) perspective, such as those in TREC in the 1990s (Over, 2001;Kelly and Lin, 2007) along with many others (Su, 1992;Dunlop, 1996;Koenemann and Belkin, 1996;Xie, 2003;Petrelli, 2008;Hearst, 2009).…”

“…In the latter, the experiment takes place in the real operational/natural setting where the IR system is typically used. There are known advantages and disadvantages and important issues acknowledged for each setting and therefore should be considered by the researcher for a careful choice (Tague-sutcliffe, 1992;Robertson, 1981;Borlund and Ingwersen, 1997;Petrelli, 2008). First, the laboratory setting offers more control of independent variables that may affect the outcomes of the experiment.…”

Methods for Evaluating Interactive Information Retrieval Systems with Users

“…Another area where there is quite a lot of systems-centered research, but not much user-centered research is cross-language retrieval, although several researchers have made contributions to this area [204,211]. Cross-language retrieval is not as widespread and common as the other types of retrieval discussed in the preceding paragraphs, so it is hard to identify search tasks and contexts.…”