Novelty-based Spatiotemporal Saliency Detection for Prediction of Gaze in Egocentric Video

Polatsek, Patrik; Benešová, Wanda; Paletta, Lucas; Perko, Roland

doi:10.1109/lsp.2016.2523339

Cited by 15 publications

(8 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Gaze data in first-person wearable systems can aid in temporal video segmentation [ 17 ], and its computational prediction has been studied [ 18 ]. The gaze data of the wearer of an egocentric camera have been used to score the importance of the frames, as the input to a fast-forward algorithm [ 19 ].…”

Section: Related Workmentioning

confidence: 99%

Glimpse: A Gaze-Based Measure of Temporal Salience

Traver

Zorío

Leiva

2021

Sensors

View full text Add to dashboard Cite

Temporal salience considers how visual attention varies over time. Although visual salience has been widely studied from a spatial perspective, its temporal dimension has been mostly ignored, despite arguably being of utmost importance to understand the temporal evolution of attention on dynamic contents. To address this gap, we proposed Glimpse, a novel measure to compute temporal salience based on the observer-spatio-temporal consistency of raw gaze data. The measure is conceptually simple, training free, and provides a semantically meaningful quantification of visual attention over time. As an extension, we explored scoring algorithms to estimate temporal salience from spatial salience maps predicted with existing computational models. However, these approaches generally fall short when compared with our proposed gaze-based measure. Glimpse could serve as the basis for several downstream tasks such as segmentation or summarization of videos. Glimpse’s software and data are publicly available.

show abstract

Section: Related Workmentioning

confidence: 99%

Glimpse: A Gaze-Based Measure of Temporal Salience

Traver

Zorío

Leiva

2021

Sensors

View full text Add to dashboard Cite

show abstract

“…Visual analysis of egocentric videos has recently became a hot research topic in computer vision [13], [14], from recognizing daily activities [2], [3] to object detection [15], video summarization [16], and predicting gaze behavior [17], [18], [19]. In the following, we review some previous work related to ours spanning Relating static and egocentric, Social interactions among egocentric viewers, and Person identification and localization.…”

Section: Related Workmentioning

confidence: 99%

Egocentric Meets Top-View

Ardeshir

Borji

2019

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Thanks to the availability of wearable devices such as GoPro cameras, smart phones, and glasses, we have now access to a plethora of videos captured from the first person perspective. Surveillance cameras and Unmanned Aerial Vehicles (UAVs) also offer tremendous amounts of video data recorded from top and oblique view points. Egocentric and surveillance vision have been studied extensively but separately in the computer vision community. The relationship between these two domains, however, remains unexplored. In this study, we make the first attempt in this direction by addressing two basic yet challenging questions. First, having a set of egocentric videos and a top-view video, does the top-view video contain all or some of the egocentric viewers? In other words, have these videos been shot in the same environment at the same time? Second, if so, how can we identify the egocentric viewers in the top-view video? These problems can become even more challenging when videos are not temporally aligned. We model each view using a graph, and compute the assignment and time-delays in an iterative-alternative fashion using spectral graph matching and time delay estimation. Such an approach handles the temporal misalignment between the egocentric videos and the top-view video.

show abstract

“…Even fewer studies supply manual annotations or develop an algorithmic detection strategy for the eye movements in this context. Saliency in 360 • [Cheng et al 2018;Nguyen et al 2018] as well as egocentric [Lee et al 2012;Li et al 2018;Polatsek et al 2016] content is gaining popularity, and this inevitably requires the collection of eye tracking data for 360 • images and videos or in the mobile eye tracking scenario. However, the data sets that are typically published provide scanpaths in the form of sequences of "fixations" [Bolshakov et al 2017;Rai et al 2017;Sitzmann et al 2018], which limits their usefulness for eye movement research.…”

Section: Data Sets and Eye Movement Annotationmentioning

confidence: 99%

“…[Lo et al 2017] only capture the head rotation data without any eye tracking, assuming that the object at the centre of the participant's field of view is the one being looked at. [Polatsek et al 2016] seem to use the term "fixation" interchangeably with "gaze point".…”

Section: Data Sets and Eye Movement Annotationmentioning

confidence: 99%

A Ground-Truth Data Set and a Classification Algorithm for Eye Movements in 360-degree Videos

Agtzidis,

Startsev,

Dorr

2019

Preprint

View full text Add to dashboard Cite

The segmentation of a gaze trace into its constituent eye movements has been actively researched since the early days of eye tracking. As we move towards more naturalistic viewing conditions, the segmentation becomes even more challenging and convoluted as more complex patterns emerge. The definitions and the well-established methods that were developed for monitor-based eye tracking experiments are often not directly applicable to unrestrained set-ups such as eye tracking in wearable contexts or with head-mounted displays. The main contributions of this work to the eye movement research for 360 • content are threefold: First, we collect, partially annotate, and make publicly available a new eye tracking data set, which consists of 13 participants viewing 15 video clips that are recorded in 360 • . Second, we propose a new two-stage pipeline for ground truth annotation of the traditional fixations, saccades, smooth pursuits, as well as (optokinetic) nystagmus, vestibuloocular reflex, and pursuit of moving objects performed exclusively via the movement of the head. A flexible user interface for this pipeline is implemented and made freely accessible for use or modification. Lastly, we develop and test a simple proof-of-concept algorithm for automatic classification of all the eye movement types in our data set based on their operational definitions that were used for manual annotation. The data set and the source code for both the annotation tool and the algorithm are publicly available at https://web.gin.g-node.org/ioannis.agtzidis/360_em_dataset.

show abstract

Novelty-based Spatiotemporal Saliency Detection for Prediction of Gaze in Egocentric Video

Cited by 15 publications

References 18 publications

Glimpse: A Gaze-Based Measure of Temporal Salience

Glimpse: A Gaze-Based Measure of Temporal Salience

Egocentric Meets Top-View

A Ground-Truth Data Set and a Classification Algorithm for Eye Movements in 360-degree Videos

Contact Info

Product

Resources

About