A user-centred evaluation framework for the Sealife semantic web browsers

Abstract

BackgroundSemantically-enriched browsing has enhanced the browsing experience by providing contextualised dynamically generated Web content, and quicker access to searched-for information. However, adoption of Semantic Web technologies is limited and user perception from the non-IT domain sceptical. Furthermore, little attention has been given to evaluating semantic browsers with real users to demonstrate the enhancements and obtain valuable feedback. The Sealife project investigates semantic browsing and its application to the life science domain. Sealife's main objective is to develop the notion of context-based information integration by extending three existing Semantic Web browsers (SWBs) to link the existing Web to the eScience infrastructure.MethodsThis paper describes a user-centred evaluation framework that was developed to evaluate the Sealife SWBs that elicited feedback on users' perceptions on ease of use and information findability. Three sources of data: i) web server logs; ii) user questionnaires; and iii) semi-structured interviews were analysed and comparisons made between each browser and a control system.ResultsIt was found that the evaluation framework used successfully elicited users' perceptions of the three distinct SWBs. The results indicate that the browser with the most mature and polished interface was rated higher for usability, and semantic links were used by the users of all three browsers.ConclusionConfirmation or contradiction of our original hypotheses with relation to SWBs is detailed along with observations of implementation issues.

Modeling User Preferences in Recommender Systems

“…However, it is paramount to draw lessons from body of research from other domains experimenting with evaluation with real-world users and systems and additional challenges identified for collecting user feedback First, users often perceive their online behaviors and preferences differently from their actual behaviors Roy et al 2010]. Additionally, live user experiments, especially in the case of a prototype or research project, often suffer from unpolished user interfaces and the propensity of bugs, which affects the user experience and hence the outcome of the evaluation [Oliver et al 2009]. It is even more challenging to evaluate recommender systems that process implicit user feedback.…”

Data Triangulation in a User Evaluation of the Sealife Semantic Web Browsers

“…A detailed breakdown of methods and results is beyond the scope of this paper, but more information can be found in [3]. This paper will focus on the aspects of the study relevant to triangulation.…”

“…Details of the questionnaires can be found in [3]; the results paint an informative picture of users' attitudes. GoPubMed/GoGene were rated the highest in the dimensions of likeability, information findability, relevance, and system speed.…”

“…In the course of the EU-funded Sealife project [2], we addressed this lack by developing an innovative framework for user-centred evaluation of Semantic Web Browsers [3] for the life sciences using data from 3 sources: web server logs, questionnaire results, and semi-structured interviews. The data provided invaluable insight into user thought processes and satisfaction with the SWBs.…”