An exploration of the digital library evaluation literature based on an ontological representation

Abstract

Evaluation is a vital research area in the digital library domain, demonstrating a growing literature in conference and journal articles. We explore the directions and the evolution of evaluation research for the period 2001-2011 by studying the evaluation initiatives presented at 2 main conferences of the digital library domain, namely the Association for Computing Machinery and the Institute of Electrical and Electronics Engineers (ACM/IEEE) Joint Conference on Digital Libraries (JCDL), and the European Conference on Digital Libraries (ECDL; since 2011 renamed to the International Conference on Theory and Practice of Digital Libraries [TPDL]). The literature is annotated using a domain ontology, named DiLEO, which defines explicitly the main concepts of the digital library evaluation domain and their correlations.The ontology instances constitute a semantic network that enables the uniform and formal representation of the critical evaluation constructs in both conferences, untangles their associations, and supports the study of their evolution. We discuss interesting patterns in the evaluation practices as well as in the research foci of the 2 venues, and outline current research trends and areas for further research.

Trends and challenges of visual search interfaces in digital libraries and repositories

“…However, the 142 studies selected did not fully discuss all the problems associated with access to digital resources in repositories; therefore, it was necessary to further classify the studies. To carry out this classification, the focus was on usability and user-centred design principles in digital libraries (Bertot et al , 2006; Chowdhury, 2002; Chowdhury et al , 2006; Heradio et al , 2012; Tsakonas and Papatheodorou, 2008; Tsakonas et al , 2013; Xie, 2006, 2008; Zhang et al , 2013). As a result, knowledge areas were established according to the main challenges, and based on these knowledge areas, a second classification was conducted related to specific problems.…”

“…These institutions deal with the storage of educational materials, such as open discovery space (Nikolas et al , 2014), Multimedia Educational Resources for Learning and Online Teaching (MERLOT) (Burns and Schell, 2002), ARIADNE (Duval et al , 2001) and Educational Repository of the Spanish Educational Community (AGREGA2) (Sarasa et al , 2009), to optimise the management of and information searching processes for their resources. However, the large number and heterogeneity of digital resources generate some limitations (Tsakonas and Papatheodorou, 2007; Tsakonas et al , 2013) that restrict the use of digital libraries and repositories of educational materials. This has led to the development and use of alternative solutions based on the implementation of vocabulary enrichment (Wang et al , 2008) and knowledge organisation systems (KOSs) (Shiri and Molberg, 2005; Solomou and Papatheodorou, 2010), which are used in classification, categorisation, linking and management of contents.…”

Evaluating digital libraries: a systematised review

“… evaluate digital libraries to identify the key user standards (Masoudnia & Ali‐Ebadi, ) analyse aspects and methods used in the evaluation of digital libraries (Azadi‐Ahmadabadi, ; Cullen, , ; Esfandiyari‐moghadam & Bayat, ; Mehrabadi, ; Noroozi, ; Saracevic, ; Zhang, ) analyse the user interface/usability of digital libraries (Adabi & Jalal‐Dizegi, ; Alipoorhafezi & Adabi, ; Frias‐Martinez, Chen & Liu, ; Jeng, ; Joo & Yeon Lee, ; Noroozi & Akbari, ) analyse search capabilities (Evaluate and compare the information displayed on digital libraries of national content consortium, ; Evaluation of search capabilities of digital libraries applications in Iran, ; Nabavi, ). efficiency of digital libraries (Evaluation of Simorgh Digital Library with emphasis on managerial perspective, ; Tramullas, Sánchez‐Casabón & Garrido‐Picazo, ) investigation of use and access capability (Carlo Bertot, Snead, Jaeger & McClure, ; Snead, Bertot, Jaeger & McClure, ) ontological aspects (Tsakonas, Mitrelis, Papachristopoulos & Papatheodorou, ; Tsakonas & Papatheodorou, ) usefulness and usability (Jabeen, Qinjian, Yihan, Jabeen & Imran, ; Quijano‐Solís & Novelo‐Peña, ; Tsakonas & Papatheodorou, ). …”

“… review (all types of review articles) (Ahmad & Abawajy, ; Alipoorhafezi & Adabi, ; Cullen, , ; Esfandiyari‐moghadam & Bayat, ; Fuhr et al., ; Jose, ; Mehrabadi, ; Noroozi, , 2010a; Noroozi & Akbari, ; Saracevic, ; Tsakonas et al., ; Tsakonas & Papatheodorou, ) checklist (Evaluate and compare the information displayed on digital libraries of national content consortium, ; Evaluation of search capabilities of digital libraries applications in Iran, ; Evaluation of Simorgh Digital Library with emphasis on managerial perspective, ; Hoe‐Lian Goh et al., ; Nabavi, ; Noroozi, 2010b; Noroozi & Montazeri, ; Tripathi & Jeevan, ; Zolghadr & Momeni, ) content analysis (Azadi‐Ahmadabadi, ; Gkoumas & Lazarinis, ; Sohrabi et al., ) online questionnaire and interview (Lai et al., ; Madle et al., ; Zhang, ) Online questionnaire (Garibay et al., ; Tsakonas & Papatheodorou, ) library methods (Retrospective studies that include reviews of articles and books, and questions from expert people) (Hassanzadeh & Sohrabzadeh, ) questionnaire and interview (Moreira et al., ; Snead et al., ) researcher developer questionnaire and checklist (mixed tools) (Amini, ; Ghaebi, Baradar & Farnaghi, ) semiexperimental method (Quijano‐Solís & Novelo‐Peña, )See Figure for the frequency of the methods used. …”

“…Taskonas applied the ontological method (Identifying the concepts of a particular domain and the relationships between these concepts) to evaluate the digital libraries in two conference papers (ACM/IEEE) and concluded that the digital library evaluation was focused on the performance, effectiveness and technical priority (Tsakonas et al., ). While the results of this review showed that the digital library evaluation was focused on service quality, thus, Taskonas results are not consistent with the present study.…”