A systematic review of interactive information retrieval evaluation studies, 1967–2006

Abstract

With the increasing number and diversity of search tools available, interest in the evaluation of search systems, particularly from a user perspective, has grown among researchers. More researchers are designing and evaluating interactive information retrieval (IIR) systems and beginning to innovate in evaluation methods. Maturation of a research specialty relies on the ability to replicate research, provide standards for measurement and analysis, and understand past endeavors. This article presents a historical overview of 40 years of IIR evaluation studies using the method of systematic review. A total of 2,791 journal and conference units were manually examined and 127 articles were selected for analysis in this study, based on predefined inclusion and exclusion criteria. These articles were systematically coded using features such as author, publication date, sources and references, and properties of the research method used in the articles, such as number of subjects, tasks, corpora, and measures. Results include data describing the growth of IIR studies over time, the most frequently occurring and cited authors and sources, and the most common types of corpora and measures used. An additional product of this research is a bibliography of IIR evaluation research that can be used by students, teachers, and those new to the area. To the authors' knowledge, this is the first historical, systematic characterization of the IIR evaluation literature, including the documentation of methods and measures used by researchers in this specialty.

Mapping recent information behavior research: an analysis of co-authorship and co-citation networks

“…However, mapping the research in IB in a way that could inform other research examining the interaction between the user, the information, and the information systems requires navigating the blurry line that separates this field from others. In that sense, it is revealing that papers such as Kelly and Sugimoto (2013), which carries out a systematic review of the area ''interactive information retrieval'' acknowledges the contributions of three different research areas: information retrieval, information behavior, and human computer interaction. Thus, exploring the common ground is inevitable from both the area of IB and IR, although the two academic communities may have different patterns of publishing, an important aspect that should be considered when interpreting results.…”

The Internationalization of the Academic Library: A Systematic Review of 25 Years of Literature on International Students

“…However, the former is focused on the analysis of research, while the latter emphasizes summarizing content. 25 The systematic review is not especially common in the LIS field, although the literature indicates that it is slowly growing in popularity. Koufogiannakis, a proponent of evidence-based practice and systematic reviews, maintains a wiki that tracks systematic reviews in the LIS literature (http://lis-systematic-reviews.wikispaces.com/).…”

Evaluating hierarchical organisation structures for exploring digital libraries

“…First, as Kelly and Sugimoto (2013) show, the number of participants in such experiments varies widely and a large number of different evaluation metrics are used, reducing the comparability of the results. Second, the complexity also means that in most cases only one or at most two hierarchies are evaluated.…”