Search result list evaluation versus document evaluation: similarities and differences

Abstract

Purpose-The purpose of this study is to compare the evaluation of search result lists and documents, in particular evaluation criteria, elements, association between criteria and elements, pre/post and evaluation activities, and the time spent on evaluation. Design/methodology/approach-The study analyzed the data collected from 31 general users through prequestionnaires, think aloud protocols and logs, and post questionnaires. Types of evaluation criteria, elements, associations between criteria and elements, evaluation activities and their associated pre/post activities, and time were analyzed based on open coding. Findings-The study identifies the similarities and differences of list and document evaluation by analyzing 21 evaluation criteria applied, 13 evaluation elements examined, pre/post and evaluation activities performed and time spent. In addition, the authors also explored the time spent in evaluating lists and documents for different types of tasks. Research limitations/implications-This study helps researchers understand the nature of list and document evaluation. Additionally, this study connects elements that participants examined to criteria they applied, and further reveals problems associated with the lack of integration between list and document evaluation. The findings of this study suggest more elements, especially at list level, be available to support users applying their evaluation criteria. Integration of list and document evaluation and integration of pre, evaluation and post evaluation activities for the interface design is the absolute solution for effective evaluation. Originality/value-This study fills a gap in current research in relation to the comparison of list and document evaluation.

Factors that influence query reformulations and search performance in health information retrieval: A multilevel modeling approach

“…Users benefit from the additional information to judge the relevance and usefulness of the search results. Additional information about the retrieved items supports users to evaluate search results more efficiently and effectively, and the result evaluation with more information would influence post‐query activities including query reformulation (Xie & Benoit, ). As users can utilize the additional information when judging the relevance or usefulness of the search results, they would be less likely to further explore additional results to check if the search results would be relevant (precision) and/or sufficient (recall) for their given task.…”

“…According to Mu et al (2014), users spend a significant amount of time interacting with these extra components in MeshMed, which eventually led to efficient search strategies with fewer query reformulation attempts. Additional information about the retrieved items supports users to evaluate search results more efficiently and effectively, and the result evaluation with more information would influence post-query activities including query reformulation (Xie & Benoit, 2013). Users with limited knowledge of medical terms tend to select incorrect terms and to adopt a completely new query based on trial and error (Vanopstal et al, 2013).…”

User involvement and system support in applying search tactics

“…In applying evaluating tactics, users focus on identifying multiple dimensions of criteria to evaluate search results or individual items, and the systems emphasize generating related information to support users' evaluation process (Xie & Benoit, 2013). The provision of document metadata, search results with short summaries, and categorized overviews facilitates users' efficient involvement in evaluating tactics (Kules & Shneiderman, 2008;Makri, Blandford, & Cox, 2008a).…”

Explore eye movement patterns in search result evaluation and individual document evaluation

“…To satisfy users’ information needs, various search engines, online databases, digital libraries are created and different search services are provided to offer approaches for information retrieval. In the entire information retrieval process, evaluation, including both search result evaluation and individual document evaluation, is one of the key activities (Xie & Benoit, ). Xie and Benoit () compared these two types of evaluation behaviors in relation to evaluation criteria, elements, evaluation activities, and so on.…”

“…In the entire information retrieval process, evaluation, including both search result evaluation and individual document evaluation, is one of the key activities (Xie & Benoit, ). Xie and Benoit () compared these two types of evaluation behaviors in relation to evaluation criteria, elements, evaluation activities, and so on. They found that although there are differences between these two types of evaluation in criteria and element aspects, they are interrelated to each other and can be transformed and integrated.…”