A comprehensive and systematic model of user evaluation of Web search engines: II. An evaluation by undergraduates

Abstract

This paper presents an application of the model described in Part I to the evaluation of Web search engines by undergraduates. The study observed how 36 undergraduate used four major search engines to find information for their own individual problems and how they evaluated these engines based on actual interaction with the search engines. User evaluation was based on 16 performance measures representing five evaluation criteria: relevance, efficiency, utility, user satisfaction, and connectivity. Non-performance (user-related) measures were also applied. Each participant searched his/ her own topic on all four engines and provided satisfaction ratings for system features and interaction and reasons for satisfaction. Each also made relevance judgements of retrieved items in relation to his/her own information need and participated in post-search interviews to provide reactions to the search results and overall performance. The study found significant differences in precision P R1 , relative recall, user satisfaction with output display, time saving, value of search results, and overall performance among the four engines and also significant engine by discipline interactions on all these measures. In addition, the study found significant differences in user satisfaction with response time among four engines, and significant engine by discipline interaction in user satisfaction with search interface. None of the four search engines dominated in every aspect of the multidimensional evaluation. Content analysis of verbal data identified a number of user criteria and users evaluative comments based on these criteria. Results from both quantitative analysis and content analysis provide insight for system design and development, and useful feedback on strengths and weaknesses of search engines for system improvement.

An overview of semantic search evaluation initiatives

“…Search tasks/topics were used at TREC and CLEF interactive tracks (Over, 1997;Gonzalo et al, 2006); while Jose et al (1998);Borlund (2000); White et al (2007) and Petrelli (2008) adopted Borlund's simulated work task. In the same context, another approach to achieve realism and motivate and engage the recruited subjects in the evaluation is to let them choose the search tasks from a pool of available tasks, for instance, in different domains (Spink, 2002;Su, 2003;White et al, 2007;Joho et al, 2008).…”

On the Improvement of Weighted Page Content Rank

“…In order to evaluate the proposed algorithm we adopted the user-based approach [9]- [11]. Many researchers adopted the user-based approach to develop effectiveness evaluation of relevancy search engine because the system-based approach ignores user's Perception, needs, and searching behavior in real-life situations.…”

User Evaluation Methods for Visual Web Search Interfaces

“…Choosing tasks from collections such as the TREC 2005 Hard Track 1 alleviates the burden of having to create difficult yet understandable tasks; however it is important to ensure that the starting query is representative of the short queries commonly used in Web search [2]. Another alternative is to have the participants use self-identified search topics [22,30,33]. However, making comparisons between the performance of participants then becomes difficult.…”

“…A method we have found to be effective is based on participants assigning relevance scores to the documents they consider [30,33], without viewing the actual documents. Measurements of task completion times are based on the participants having identified a pre-determined number of relevant documents (e.g., 10 relevant documents).…”

“…By comparing the relevance scores of the participants to ground truth relevance scores assigned by a panel of experts, measurements such as error rates and accuracy can be calculated. Others have suggested allowing participants to continue searching until they are satisfied [2,30,33] or to limit the number of documents that can be considered [22,36]; however these techniques make it difficult to measure the time to task completion.…”