The PRET A Rapporter framework: Evaluating digital libraries from the perspective of information work

Abstract

.ac.uk, tel. +44 20 7679 5288, fax +44 7679 5295The strongest tradition of IR systems evaluation has focused on system effectiveness; more recently, there has been a growing interest in evaluation of Interactive IR systems, balancing system and user-oriented evaluation criteria. In this paper we shift the focus to considering how IR systems, and particularly digital libraries, can be evaluated to assess (and improve) their fit with users' broader work activities. Taking this focus, we answer a different set of evaluation questions that reveal more about the design of interfaces, user-system interactions and how systems may be deployed in the information working context. The planning and conduct of such evaluation studies share some features with the established methods for conducting IR evaluation studies, but come with a shift in emphasis; for example, a greater range of ethical considerations may be pertinent. We present the PRET A Rapporter framework for structuring user-centred evaluation studies and illustrate its application to three evaluation studies of digital library systems.

Methods for Evaluating Interactive Information Retrieval Systems with Users

“…Additional attention is given to the study of different types of IIR systems in the final section of this paper. Digital libraries, a specific setting where IIR occurs, are also not discussed explicitly, but again, much of the material in this paper will be relevant to those working in this area [29].…”

Metodologia para avaliação do nível de usabilidade de bibliotecas digitais: um estudo na Biblioteca Virtual de Saúde

“…Para Blandford et al (2008), muitos estudos publicados sobre avaliação de bibliotecas digitais são relatos de avaliações de sistemas específicos, envolvendo testes com usuários ou avaliação de especialistas. Esses estudos de avaliação podem ser baseados em análise quantitativa, como os que envolvem o uso de logs de transação, e qualitativos, como as técnicas de entrevista, observação ou o think aloud (pensar em voz alta).…”

Untitled

“…Nevertheless, it is important to ensure that, changes in design at each stage are considered, designed, and recorded to provide a clear audit trail of the final methodology adopted. Blandford et al (2008) suggested that PRETAR is an improvement over Rogers et al's (2011) DECIDE approach for structuring user-centered evaluative studies. Their criticism of DECIDE is that its steps are interdependent and can confuse, whereas, PRETAR's are not.…”

“…Yet both frameworks aim to reveal more about design/evaluation studies than do standard approaches. Furthermore, Blandford et al (2008) commented that PRETAR can be used for planning, conducting, and discussing studies; in other words, for the full cycle of a field study. The PRETAR framework is presented as a sequential model, although comments in Blandford et al (2008) acknowledge that ethical issues, for instance, can impinge on planning data collection and analysis, which implies that there is still some overlap.…”

“…PRETAR was developed to explicitly structure user-centered evaluation studies to ensure that the necessary elements for such studies are individually and independently considered and presented in a logical manner (Blandford et al, 2008). However, our research of the literature to determine the value of the framework failed to reveal any studies other than those by Blandford and her colleagues (Blandford et al, 2008;Makri, Blandford, & Cox, 2011). Therefore, this paper is motivated by a desire to independently evaluate the usefulness of the PRETAR framework.…”