Search strategies in multimodal image retrieval

Abstract

This paper reports on a study on search strategies in multimodal image retrieval. We analyzed the queries and search tactics employed by image journalism professionals and nonprofessionals in a user test. Transaction log data show that searchers are able to combine up to four query modes into a query. Most queries combined at least two of the modes (text, color, sketch, quality, and category). Task type was shown to affect the choice of which modes to employ. Known item and data search tasks led to queries combining text, color and category modes. Visually cued tasks resulted in searches combining several content-based and textual modes. Conceptual tasks led to a large number of queries by text and category only. User background also significantly affected the types of queries constructed. Professionals used the color mode more whereas nonprofessionals drew more sketches. Non-professionals were more likely to switch query modes whereas professionals edited the content of their queries. We described the search processes with Markov models and maximal repeating patterns. Common patterns and probable transitions dealt with querying, inspecting result images and saving them into the workspace or iterating queries of the same type. The results indicate a need to support multimodal image query formulation.

Investigating variation in querying behavior for image searches on the web

“…However, it is interesting to note that the average number of query iterations during a session (11.24 queries) in this study is much higher than those in the previous studies of textual search sessions, e.g., 7.7 queries in Aula and Käki (2003)'s study and 7.8 queries in Toms (2008) study. Also, image searches in Westman, Lustila, and Oittinen (2008)'s study yielded an average number of queries issued in a task of 5.6 queries for professionals and 4.9 for non‐professionals).…”

Visual information seeking

“…User experiences with different versions of the experimental systems were assessed. Westman, Lustila, and Oittinen (), similarly, employed image journalism professionals with years of experience in the magazine industry and other nonprofessional users, or those who had no experience with the professional use of digital multimedia. Both groups of users were asked to conduct a range of individual image search tasks, spanning needs for specific objects, general colors and color schemes, physical qualities (e.g., size and resolution), and those depicting abstract or interpretative (e.g., “hurry” and “quest for success”) meaning.…”

“…Westman et al. () measured interface use and queries, and surveyed users about usability and learnability of the experimental system, which included their impressions of its different features. Quantitative comparisons between the different factors and variables were performed, where the objective was to produce further evidence about the use of content‐based (e.g., visual) search features for retrieving images, from an end‐user point‐of‐view, when used with text in one multimodal search interface.…”

“…Westman et al. (), in an experimental video retrieval study, further demonstrated the importance of scoping search queries and strategies by observing how 84.5% of queries were ultimately modified by users. Further, query modification, as a search action, has been shown to be a regular occurrence in an image retrieval context (Jörgensen & Jörgensen, ).…”

QUEST: Towards a Multi-modal CBIR Framework Combining Query-by-Example, Query-by-Sketch, and Text Search

“…Integrating text search with either query-by-sketch or queryby-example is the most common. Studies [16] have shown that there is user ability and willingness to combine different search modalities in image retrieval. In this paper, we extend the integration of multiple modes of images retrieval to build on the strengths of every individual mode.…”