Task-Based Information Interaction Evaluation

Abstract

Evaluation is central in research and development of information retrieval (IR). In addition to designing and implementing new retrieval mechanisms, one must also show through rigorous evaluation that they are effective. A major focus in IR is IR mechanisms' capability of ranking relevant documents optimally for the users, given a query. Searching for information in practice involves searchers, however, and is highly interactive. When human searchers have been incorporated in evaluation studies, the results have often suggested that better ranking does not necessarily lead to better search task, or work task, performance. Therefore, it is not clear which system or interface features should be developed to improve the effectiveness of human task performance. In the present article, we focus on the evaluation of task-based information interaction (TBII). We give special emphasis to learning tasks to discuss TBII in more concrete terms. Information interaction is here understood as behavioral and cognitive activities related to task planning, searching information items, selecting between them, working with them, and synthesizing and reporting. These five generic activities contribute to task performance and outcome and can be supported by information systems. In an attempt toward task-based evaluation, we introduce program theory as the evaluation framework. Such evaluation can investigate whether a program consisting of TBII activities and tools works and how it works and, further, provides a causal description of program (in)effectiveness. Our goal in the present article is to structure TBII on the basis of the five generic activities and consider the evaluation of each activity using the program theory framework. Finally, we combine these activity-based program theories in an overall evaluation framework for TBII. Such an evaluation is complex due to the large number of factors affecting information interaction. Instead of presenting tested program theories, we illustrate how the evaluation of TBII should be accomplished using the program theory framework in the evaluation of systems and behaviors, and their interactions, comprehensively in context.

Search task features in work tasks of varying types and complexity

“…Several researchers highlight the importance of taskbased information searching (TBIS) as a research area (J€ arvelin et al, 2015;Kelly, Arguello, & Capra, 2013;Vakkari, 2003). However, naturalistic field studies of TBIS are rare.…”

Evaluation

“…User-oriented evaluation: approaches for evaluation that involves users in some way typically fall within the area of Interactive IR [24,8,23]. Criteria used to assess the system are typically concerned with how well users achieve their goals or tasks and their success and satisfaction with the outputs.…”

Identification of information search strategies and their impact on learning outcome in learning‐related tasks

“…However, learning-related search is more complex exploratory search, rather than just retrieving specific factual information (Marchionini, 2006). When searching for learning-related tasks, people may involve in various activities, such as task planning, searching information items, selecting information items, working with information items, and synthesizing and reporting (Järvelin et al, 2015). Therefore, it is necessary to examine what information search strategies users adopt when searching for learning-related tasks, based on users' interactions on various types of web pages.…”