Evaluation is central in research and development of information retrieval (IR). In addition to designing and implementing new retrieval mechanisms, one must also show through rigorous evaluation that they are effective. A major focus in IR is IR mechanisms' capability of ranking relevant documents optimally for the users, given a query. Searching for information in practice involves searchers, however, and is highly interactive. When human searchers have been incorporated in evaluation studies, the results have often suggested that better ranking does not necessarily lead to better search task, or work task, performance. Therefore, it is not clear which system or interface features should be developed to improve the effectiveness of human task performance. In the present article, we focus on the evaluation of task-based information interaction (TBII). We give special emphasis to learning tasks to discuss TBII in more concrete terms. Information interaction is here understood as behavioral and cognitive activities related to task planning, searching information items, selecting between them, working with them, and synthesizing and reporting. These five generic activities contribute to task performance and outcome and can be supported by information systems. In an attempt toward task-based evaluation, we introduce program theory as the evaluation framework. Such evaluation can investigate whether a program consisting of TBII activities and tools works and how it works and, further, provides a causal description of program (in)effectiveness. Our goal in the present article is to structure TBII on the basis of the five generic activities and consider the evaluation of each activity using the program theory framework. Finally, we combine these activity-based program theories in an overall evaluation framework for TBII. Such an evaluation is complex due to the large number of factors affecting information interaction. Instead of presenting tested program theories, we illustrate how the evaluation of TBII should be accomplished using the program theory framework in the evaluation of systems and behaviors, and their interactions, comprehensively in context.

In task-based information searching, the task at hand is a central factor affecting information search. Task complexity, in particular, has been discovered to affect searching. In the present study, we shadowed the tasks of seven people working in city administration. The data consist of shadowing field notes, voice recordings, photographs and forms. We study, how task complexity affects information searching and information resource use. Task complexity was defined through the task performer's own experience (perceived task complexity) and her estimates of her a priori knowledge concerning the task. We analyzed the data both qualitatively and quantitatively, focusing on the links between task complexity and the use of information resources, information searching and problems encountered. We found that task complexity has a central but ambiguous relationship to task performance. The clearest differences were found between simple and complex tasks. In addition, perceived task complexity seems to affect the ways of performing the task more than a priori knowledge. The more complex a task is perceived, the more searches are performed and the more they concentrate on networked resources instead of information systems provided by the organization (SPOs). The use of resources on the task performer's PC and the SPOs decreases when complexity increases. In proportion, the use of networked resources and communication resources increases. The total number of information resources used is somewhat greater in complex and semi-complex tasks than in simple tasks; and each resource is used for a longer time on average. Our study shows that task context and especially task complexity seems to affect information searching and the selection of sources.

Purpose The purpose of this paper is to investigate information retrieval (IR) in the context of authentic work tasks (WTs), as compared to traditional experimental IR study designs. Design/methodology/approach The participants were 22 professionals working in municipal administration, university research and education, and commercial companies. The data comprise 286 WTs and 420 search tasks (STs). The data were collected in natural situations. It includes transaction logs, video recordings, interviews, observation, and daily questionnaires. Findings The analysis included the effects of WT type and complexity on the number of STs, queries, search keys and types of queries. The findings suggest that simple STs are enough to support most WTs. Complex WTs (vs more simple ones) and intellectual WTs (vs communication, support and editing WTs) include more STs than other WT categories. Research limitations/implications Further research should address the problems related to controllability of field studies and enhance the use of realistic WT situations in test-based studies, as well. Originality/value The study is an attempt to bring traditional IR studies and realistic research settings closer to each other. Using authentic WTs when studying IR is still rare. The representativeness of the WT/ST types used in interactive IR experiments should be carefully addressed: in the work flow, people seldom consciously recognise separate “STs”. This means that STs may mainly be an academic construct even to the point that studying IR without a decent context does violence to the further understanding of the phenomenon.

The aim of the study was to test whether query expansion by approximate string matching methods is beneficial in retrieval from historical newspaper collections in a language rich with compounds and inflectional forms (Finnish). First, approximate string matching methods were used to generate lists of index words most similar to contemporary query terms in a digitized newspaper collection from the 1800s. Top index word variants were categorized to estimate the appropriate query expansion ranges in the retrieval test. Second, the effectiveness of approximate string matching methods, automatically generated inflectional forms, and their combinations were measured in a Cranfield-style test. Finally, a detailed topic-level analysis of test results was conducted. In the index of historical newspaper collection the occurrences of a word typically spread to many linguistic and historical variants along with optical character recognition (OCR) errors. All query expansion methods improved the baseline results. Extensive expansion of around 30 variants for each query word was required to achieve the highest performance improvement. Query expansion based on approximate string matching was superior to using the inflectional forms of the query words, showing that coverage of the different types of variation is more important than precision in handling one type of variation. IntroductionDigitization is a good way to preserve cultural heritage documents and make them widely accessible for researchers and the general public. Cultural institutions are aware of this potential and often consider digitization of their cultural heritage collections as an obligation. Consequently, the quantity of digitized historical documents available is constantly growing. Transforming print cultural heritage collections into digital resources accessible and searchable through modern information and communication technologies requires that the digitized document images are transformed into digital text through optical character recognition (OCR). While OCR can currently reach over 99% accuracy in recognition of characters from high-quality images of original documents with a simple book layout, the accuracy for historical newspapers is lower than that. OCR quality is dependent on the environment and the condition of the original documents: print and paper quality, typefaces, and layout complexity affect the accuracy of the result. Generally, the older the newspaper is, the lower the accuracy rate is likely to be. Holley (2009) reported raw character recognition accuracy rates varying from 71% to 98% in a sample of digitized newspapers from 1803-1954, the lowest rate indicating almost every third character being erroneously recognized and virtually all words containing errors. Even a 98% accuracy rate results in an error in, on average, every sixth word in Finnish text (with an average word length of around eight characters), if the errors are evenly distributed. Such error rates may lead to a quadrupling of the number of unique index words and sign...

Information seeking research often reports about types of information resources, ways of acquiring them and opinions on their importance in various professions. Based on self-reporting, these findings are affected by human memory and rationalisation. This article proposes a new way of studying information resource use – based on dwell time in the context provided by concrete work tasks. We use log data of 21 information workers from six organisations to analyse how work task complexity is connected to the time used in various information resources; how task complexity is connected to information resource use in different task types. Unlike traditionally, our findings consist of objective data on which resource types are used, and for how long, in work tasks of varying complexity and type. For example, the findings suggest that growing work task complexity increases the dwell time in local personal computer (PC) resources; these resources are especially popular in intellectual tasks. Such findings help understand factors affecting information resource use. Likewise, they help focus attention on most time-consuming aspects of task-based information interaction when developing support for work.