Grid-based Evaluation Metrics for Web Image Search

Abstract

Compared to general web search engines, web image search engines display results in a di erent way. In web image search, results are typically placed in a grid-based manner rather than a sequential result list. In this scenario, users can view results not only in a vertical direction but also in a horizontal direction. Moreover, pagination is usually not (explicitly) supported on image search search engine result pages (SERPs), and users can view results by scrolling down without having to click a "next page" button. These di erences lead to di erent interaction mechanisms and user behavior patterns, which, in turn, create challenges to evaluation metrics that have originally been developed for general web search. While considerable e ort has been invested in developing evaluation metrics for general web search, there has been relatively little e ort to construct grid-based evaluation metrics. To inform the development of grid-based evaluation metrics for web image search, we conduct a comprehensive analysis of user behavior so as to uncover how users allocate their attention in a grid-based web image search result interface. We obtain three ndings: (1) "Middle bias": Con rming previous studies, we nd that image results in the horizontal middle positions may receive more attention from users than those in the leftmost or rightmost positions. (2) "Slower decay": Unlike web search, users' attention does not decrease monotonically or dramatically with the rank position in image search, especially within a row. (3) "Row skipping": Users may ignore particular rows and directly jump to results at some distance. Motivated by these observations, we propose corresponding user behavior assumptions to capture users' search interaction processes and evaluate their search performance. We show how to derive new metrics from these assumptions and demonstrate * Corresponding author This paper is published under the Creative Commons Attribution 4.0 International (CC-BY 4.0) license. Authors reserve their rights to disseminate the work on their personal and corporate Web sites with the appropriate attribution.

Preference-based Evaluation Metrics for Web Image Search

“…On the basis of absolute relevance judgments, traditional relevancebased evaluation metrics have been used to generate a ranking score for a given result list [28,30], e.g., normalized discounted cumulative gain (NDCG) [10] and rank-biased precision (RBP) [13]. Furthermore, Xie et al [27] propose grid-based evaluation metrics by considering grid information to revise traditional list-based metrics.…”

“…Specifically, image quality depicts the artistic value of a given image, which is a vague concept and hard to define in finegrained grades. (3) Existing work shows that relevance-based evaluation metrics do not correlate well with user satisfaction in web image search [27] and that they do not distinguish well between different systems or ranking functions [26]. Preference judgments have been investigated as an alternative to absolute judgments in general web search [1,8,20].…”

Modeling User Behavior for Vertical Search: Images, Apps and Products

“…• Introduce previous work on evaluation metrics for vertical search, including offline metrics and online metrics [20,27,29]. [10 minutes] Conclusion and future directions.…”

“…For example, Xie et al [23] propose an image search intent taxonomy consisting of three classes: "Explore/Learn", "Entertain", "Locate/Acquire" and Su et al [19] categorize search intents of product search users into Target Finding (TF), Decision Making (DM) and Exploration (EP). Moreover, result placement and interaction mechanisms of vertical search have been studied [20,27].…”

“…For instance, Xie et al [25] utilize user behavior, i.e., click and cursor hovering, in a search session to capture users' short-term visual preference in image search. For evaluation metrics, different types of result placement and unique user behavior have been considered [20,27,31]. Besides topical relevance, aspects such as image quality (image search), revenue (product search) and download (app search) need to be optimized.…”