Developing a holistic model for digital library evaluation

Abstract

The objective of the research is to develop a holistic model for digital library (DL) evaluation. To develop such a model, a three-stage research approach was applied: exploration, confirmation, and verification. During the exploration stage, a literature review was conducted, and then an interview along with card sorting technique was employed to collect perceptions from DL experts with emphasis on determining what criteria should be used in DL evaluation. Then, the criteria identified from the exploration were used for developing an online survey during the confirmation stage. Heterogeneous DL stakeholders were asked to rate the importance of each criterion to DL evaluation.The holistic model was constructed by utilizing descriptive and inference statistical techniques. Its holistic nature was ensured through: (1) incorporation of various DL stakeholders' perspectives in light of Marchionini's multifaceted evaluation approach, and (2) inclusion of all digital library levels suggested by Saracevic's stratified information retrieval model. Eventually, in the verification stage, selected criteria from the model were tested in real DL use setting. Some significant findings include: (1) consistently perceived important criteria for DL evaluation. DL stakeholders care more about premise (e.g., accessibility and iii iii sustainability of a DL), process (e.g., ease of use, technology reliability, and service responsiveness), and direct performance (e.g., usefulness of information, successfulness and efficiency of task completion), whereas less concerned about indirect factors (e.g., personalization, behavior change, service courtesy, and extended social effects); (2) inter-group divergence in importance perception for some evaluation criteria. The divergence primarily exists between the user and other DL stakeholder groups; (3) some promising criteria (e.g., comprehensiveness of collection, integrity of information, integration of service to information seeking path, collaboration/sharing) augment the existing DL evaluations whereby important criteria have essentially been covered; and (4) most importantly, the core dissertation objective is fulfilled, that is the construction of the holistic evaluation model, in which heterogeneous stakeholders' perspectives at all DL levels are presented. Moreover, some prestigious journals also have devoted special issues to this subject (e.g., 2 Science 1993Science , 2000 Information Processing and Management, 1999; Library Trends, 2000; Communication of ACM, 1995. IEEE Computer, 1995. Journal of American Society for InformationHowever, behind this fast-growing scene are some weaknesses that might hinder the progress of DL innovation. One of the remarkable weaknesses is reflected in the two competing visions of what is a DL, which are pinpointed by Borgman in 1999. One vision, with the research domain (e.g., Fox et al., 1993;Lesk, 1997) Unsurprisingly, the divergent visions of the definition of the DL cannot but yield to different foci in DL development. The view ...

Signals and Signposts of the Future: Literary Festival Consumption in 2050

“…First of these are Librarians, who are likely to work as latter-day DJs able to set up wherever required, already supporting and enhance the literary convergences of academics, co-learners (students and teaching staff) information professionals and -of course -authors and readers. Their work looks to both digitize and synthesize information for virtual environments (Leonard and Morasch 2012;Places et al 2007;Zhang 2010). As enablers, they too will have a new importance in the literary event future.…”

How are we Measuring Up? Evaluating Research Data Services in Academic Libraries

“…Chapman, DeRidder, and Thompson (2015) note that generalizability for evaluating digital libraries across institutions is difficult due to wide differences in software and platforms. In digital library evaluation, there has been a heavy focus on analytics, costs, user studies, the user interface, software, and citations (Hufford, 2013;Saracevic, 2000;Zhang, 2010). Monitoring usage trends for complex services and infrastructure is more difficult.…”

Testing a Multidimensional and Hierarchical Quality Assessment Model for Digital Libraries

“…In the context of Romania, no studies were found addressing this topic. The overall quality of DLs is insufficiently studied and reported [44], and DL quality and evaluation is a very underrepresented research area in the digital library literature [13]. While previous studies have provided useful and interesting information regarding the topic, this study proposes and tests a multidimensional and hierarchical model which can be used to assess the quality of digital libraries.…”

“…In line with the growing number of operational DLs, there is also growing interest among researchers to investigate the quality of DLs [1], [13], [22], [34], [42], [44]. In the context of Romania, no studies were found addressing this topic.…”

“…While previous studies have provided useful and interesting information regarding the topic, this study proposes and tests a multidimensional and hierarchical model which can be used to assess the quality of digital libraries. The model is based on Zhang model [44] and DeLone & McLean IS success model [9]. This paper is organized as follows.…”