Eye tracking assisted extraction of attentionally important objects from videos

Karthikeyan, S.; Ngo, Thuyen; Eckstein, Miguel P.; Manjunath, B.S.

doi:10.1109/cvpr.2015.7298944

Cited by 27 publications

(14 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since eye gaze is known to provide cues on the underlying cognitive processes [ 11 , 12 ], it can be expected to be particularly useful for this kind of video-processing task. However, despite being used in some computer vision problems [ 13 , 14 , 15 , 16 ], its general use has been limited.…”

Section: Related Workmentioning

confidence: 99%

Glimpse: A Gaze-Based Measure of Temporal Salience

Traver

Zorío

Leiva

2021

Sensors

View full text Add to dashboard Cite

Temporal salience considers how visual attention varies over time. Although visual salience has been widely studied from a spatial perspective, its temporal dimension has been mostly ignored, despite arguably being of utmost importance to understand the temporal evolution of attention on dynamic contents. To address this gap, we proposed Glimpse, a novel measure to compute temporal salience based on the observer-spatio-temporal consistency of raw gaze data. The measure is conceptually simple, training free, and provides a semantically meaningful quantification of visual attention over time. As an extension, we explored scoring algorithms to estimate temporal salience from spatial salience maps predicted with existing computational models. However, these approaches generally fall short when compared with our proposed gaze-based measure. Glimpse could serve as the basis for several downstream tasks such as segmentation or summarization of videos. Glimpse’s software and data are publicly available.

show abstract

Section: Related Workmentioning

confidence: 99%

Glimpse: A Gaze-Based Measure of Temporal Salience

Traver

Zorío

Leiva

2021

Sensors

View full text Add to dashboard Cite

show abstract

“…People usually use gaze as an extra information channel for solving computer vision problems, such as action recognition [25,26], object detection [27,24], saliency prediction [28,29] and segmentation [30,16]. In video analysis, since subjects tend to watch at the moving objects, gaze are also widely used to localize important objects [31,32,33,34].…”

Section: Eye-tracking Gazementioning

confidence: 99%

Gaze latent support vector machine for image classification improved by weakly supervised region selection

Wang

Thome

Cord

2017

Pattern Recognition

View full text Add to dashboard Cite

HAL is a multidisciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

show abstract

“…The correlation between fixations and objects is also becoming important in the context of saliency prediction [24], [25]. There have been several methods proposed for robust gaze-guided object segmentation [12], [26], [27], and based on a similar assumption fixation information is also used for localizing important objects in videos [5], [28], [29]. While these works mainly focused on object localization and the recognition was assumed to be purely vision-based, there have been several attempts to further use gaze information for object recognition.…”

Section: Gaze For Object Localization and Recognitionmentioning

confidence: 99%

“…Human gaze reflects processes of cognition and perception and therefore represents a rich source of information about the observer of a visual scene. Consequently, gaze has successfully been used for tasks such as eye-based user modeling [1], [2], [3], [4], [5] and opened up new opportunities to further advance human-machine collaborationcollaborative human-machine vision systems in which part of the processing is carried out by the machine while another part is performed by a human and conveyed to the machine via gaze information.…”

Section: Introductionmentioning

confidence: 99%

“…For these reasons, gazeenabled computer vision methods have been attracting increasing interest in recent years. Prior work has typically focused on object-centric tasks, such as object localization [5], [12] or recognition [3], [13]. Although gaze information is potentially even richer for tasks that require holistic scene understanding [14], [15], the integration of gaze information for scene-centric computer vision algorithms, such as for image captioning, has not yet been explored.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Seeing with Humans: Gaze-Assisted Neural Image Captioning

Sugano,

Bulling

2016

Preprint

View full text Add to dashboard Cite

Gaze reflects how humans process visual scenes and is therefore increasingly used in computer vision systems. Previous works demonstrated the potential of gaze for object-centric tasks, such as object localization and recognition, but it remains unclear if gaze can also be beneficial for scene-centric tasks, such as image captioning. We present a new perspective on gaze-assisted image captioning by studying the interplay between human gaze and the attention mechanism of deep neural networks. Using a public large-scale gaze dataset, we first assess the relationship between state-of-the-art object and scene recognition models, bottom-up visual saliency, and human gaze. We then propose a novel split attention model for image captioning. Our model integrates human gaze information into an attention-based long short-term memory architecture, and allows the algorithm to allocate attention selectively to both fixated and non-fixated image regions. Through evaluation on the COCO/SALICON datasets we show that our method improves image captioning performance and that gaze can complement machine attention for semantic scene understanding tasks.

show abstract

Eye tracking assisted extraction of attentionally important objects from videos

Abstract: Visual attention is a crucial indicator of the relative importance of objects in visual scenes

Cited by 27 publications

References 42 publications

Glimpse: A Gaze-Based Measure of Temporal Salience

Glimpse: A Gaze-Based Measure of Temporal Salience

Gaze latent support vector machine for image classification improved by weakly supervised region selection

Seeing with Humans: Gaze-Assisted Neural Image Captioning

Contact Info

Product

Resources

About