Categorical relevance judgment

Abstract

In this study we aim to explore users' behaviour when assessing search results relevance based on the hypothesis of categorical thinking. In order to investigate how users categorise search engine results, we perform several experiments where users are asked to group a list of 20 search results into a number of categories, while attaching a relevance judgment to each formed category. Moreover, to determine how users change their minds over time, each experiment was repeated three times under the same conditions, with a gap of one month between rounds. The results show that on average users form 4-5 categories. Within each round the size of a category decreases with the relevance of a category. To measure the agreement between the search engine's ranking and the users' relevance judgments, we defined two novel similarity measures, the average concordance and the MinMax swap ratio. Similarity is shown to be the highest for the third round as the users' opinion stabilises. Qualitative analysis uncovered some interesting points, in particular, that users tended to categorise results by type and reliability of their source, and particularly, found commercial sites less trustworthy, and attached high relevance to Wikipedia when their prior domain knowledge was limited.

Evaluating Local Multilingual Health Care Information Environments on the Internet: A Pilot Study

“…Six search strings were used, each prefecture’s name ( n = 47) and one of the following health care information terms: “health system”, “hospital list”, “hospitals”, “emergency services”, “medical interpreters”, or “national health insurance.” These 282 search string combinations mimicked typical search query syntax [ 28 ]. The first three pages of search results (30 links) were assessed for inclusion with each search [ 29 ]; 8460 links were reviewed in total.…”

Financial literacy resources in US public libraries: website analysis

“…Most of the research in this area focuses on the relevancy of search engine results and has shown that individuals' assessments of relevancy tend to change over time. One common alternative to individual assessments of relevancy is to rely on the "wisdom of crowds" either through user studies or via click-through rates, which appears to improve the stability of relevancy judgments (Zhitomirsky-Geffet et al, 2016;Zhitomirsky-Geffet et al, 2018). Studies evaluating the information retrieval of library discovery tools have often relied on user observation in laboratory settings, log analysis or comparison of search results to student citations in particular projects or courses (Behnert and Lewandowski, 2017;Galbreath et al, 2021).…”

Sentiment classification in social media data by combining triplet belief functions

“…For polarity classification, we convert the labeled data into three categories as done by Pang et al (2002). Like the categorical judgment given by Zhitomirsky‐Geffet, Bar‐Ilan, and Levene (2018), the reviews labeled with 1 and 2 stars are considered very negative and negative, whereas reviews with 4 and 5 stars are considered positive and very positive, and reviews with 3 stars are regarded as neutral (Pang et al, 2002). For this study, we confine our focus only on discriminating between positive and negative polarity sentiment.…”