Can relevance of images be inferred from eye movements?

Abstract

Query formulation and efficient navigation through data to reach relevant results are undoubtedly major challenges for image or video retrieval. Queries of good quality are typically not available and the search process needs to rely on relevance feedback given by the user, which makes the search process iterative. Giving explicit relevance feedback is laborious, not always easy, and may even be impossible in ubiquitous computing scenarios. A central question then is: Is it possible to replace or complement scarce explicit feedback with implicit feedback inferred from various sensors not specifically designed for the task? In this paper, we present preliminary results on inferring the relevance of images based on implicit feedback about users' attention, measured using an eye tracking device. It is shown that, in reasonably controlled setups at least, already fairly simple features and classifiers are capable of detecting the relevance based on eye movements alone, without using any explicit feedback

Image ranking with implicit feedback from eye movements

“…Eye movements can be treated as an implicit relevance feedback when the user is not consciously aware of their eye movements being tracked. Eye movement as implicit feedback has recently been used in the image retrieval setting [Oyekoya and Stentiford 2007;Klami et al 2008;Pasupa et al 2009]. [Oyekoya and Stentiford 2007;Klami et al 2008] used eye movements to infer a binary judgement of relevance while [Pasupa et al 2009] makes the task more complex and realistic for searchbased task by asking the user to give multiple judgement of relevance.…”

“…Eye movement as implicit feedback has recently been used in the image retrieval setting [Oyekoya and Stentiford 2007;Klami et al 2008;Pasupa et al 2009]. [Oyekoya and Stentiford 2007;Klami et al 2008] used eye movements to infer a binary judgement of relevance while [Pasupa et al 2009] makes the task more complex and realistic for searchbased task by asking the user to give multiple judgement of relevance. Furthermore, earlier studies of Hardoon et al [2007] and Ajanki et al [2009] explored the problem of where an implicit information retrieval query is inferred from eye movements measured during a reading task.…”

Identifying Objects in Images from Analyzing the Users’ Gaze Movements for Provided Tags

“…In information retrieval, several approaches use eye-tracking to identify images in a search result as attractive or important and use this information as implicit user feedback to improve the image search, e.g., [6,2,5]. Jaimes et al [3] carried out a preliminary analysis of identifying common gaze trajectories in order to classify images into five, predefined semantic categories.…”

“…We use two baselines that have been applied to evaluate relevance feedback from gaze information in [6] and [5]. We compare the precision P for image-tag-pair assignments calculated from the baseline "naive" (a) and the baseline "random" (b) with the mere measure meanVisitDuration (c) and the meanVisitDuration measure including region extension and weighting (d).…”

Learning to rank images from eye movements

“…Eye movements as implicit feedback has recently been considered in the text retrieval setting [11,4,1]. To the best of our knowledge however, at the time of writing, only [10,8] used eye movements for image retrieval. They only infer a binary judgement of relevance whereas in our experiments, we make the task more complex and realistic for search-based tasks by asking the user to rank a set of images on a screen in order of relevance to a specific topic while the eye movements are recorded.…”