DGaze: CNN-Based Gaze Prediction in Dynamic Scenes

Li, Sheng; Zhang, Congyi; Yi, Kangrui; Wang, Guoping; Manocha, Dinesh

doi:10.1109/tvcg.2020.2973473

Cited by 60 publications

(29 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, these might be put into question out of the lab and in dynamic environments. One solution could be that of a data-driven strategy (Le Meur and Coutrot, 2016;Hu et al, 2020), albeit raising in turn the problem of generalizability. Further, the accumulator model lacks of a detailed account for the actual handling of within-patch items (i.e., what would be considered "prey handling" in the animal ecology field).…”

Section: Discussionmentioning

confidence: 99%

Gazing at Social Interactions Between Foraging and Decision Theory

D’Amelio

Boccignone

2021

Front. Neurorobot.

View full text Add to dashboard Cite

Finding the underlying principles of social attention in humans seems to be essential for the design of the interaction between natural and artificial agents. Here, we focus on the computational modeling of gaze dynamics as exhibited by humans when perceiving socially relevant multimodal information. The audio-visual landscape of social interactions is distilled into a number of multimodal patches that convey different social value, and we work under the general frame of foraging as a tradeoff between local patch exploitation and landscape exploration. We show that the spatio-temporal dynamics of gaze shifts can be parsimoniously described by Langevin-type stochastic differential equations triggering a decision equation over time. In particular, value-based patch choice and handling is reduced to a simple multi-alternative perceptual decision making that relies on a race-to-threshold between independent continuous-time perceptual evidence integrators, each integrator being associated with a patch.

show abstract

Section: Discussionmentioning

confidence: 99%

Gazing at Social Interactions Between Foraging and Decision Theory

D’Amelio

Boccignone

2021

Front. Neurorobot.

View full text Add to dashboard Cite

show abstract

“…On the other hand, the ecology of animal movements is still struggling on the point [15] in spite of an important body of research laid down over years. One solution could be that of a data-driven strategy [45], [50], albeit raising in turn the problem of generalisability.…”

Section: Discussionmentioning

confidence: 99%

“…As to variability, when looking at natural images, movies [44], or even dynamic virtual reality scenes [45] under a freeviewing or a general-purpose task, there is a small probability that two observers will fixate exactly the same location at exactly the same time. Such variations in individual scan paths (as regards chosen fixations, spatial scanning order, and fixation duration) still hold when the scene contains semantically rich "objects" and can become idiosyncratic [20].…”

Section: ) Problems With W: Levels Of Representationmentioning

confidence: 99%

On Gaze Deployment to Audio-Visual Cues of Social Interactions

et al. 2020

View full text Add to dashboard Cite

Attention supports our urge to forage on social cues. Under certain circumstances, we spend the majority of time scrutinising people, markedly their eyes and faces, and spotting persons that are talking. To account for such behaviour, this paper develops a computational model for the deployment of gaze within a multimodal landscape, namely a conversational scene. Gaze dynamics is derived in a principled way by reformulating attention deployment as a stochastic foraging problem. Model simulation experiments on a publicly available dataset of eye-tracked subjects are presented. Results show that the simulated scan paths exhibit similar trends of eye movements of human observers watching and listening to conversational clips in a free-viewing condition.

show abstract

“…In [12] a novel CNN is proposed for gaze prediction based on combining object position, head velocity, and saliency features. The authors conducted their experimental dataset composed of 43 users for dynamic scenes.…”

Section: Gaze Prediction In Egocentric Videosmentioning

confidence: 99%

“…Without knowledge about the field of view (FoV), 360 • highdefinition video applications require streamed full panorama scenes that contain both visible and invisible parts of the FoV, thus overly straining the bandwidth-limited network [10]. In addition, real-time field of view (FoV) extraction, such as saliency maps, image flow, and gaze history for gaze prediction, easily exceeds the processing resources of mobile hardware, such as VR HMDs or mobile phones [8,11,12].…”

Section: Introductionmentioning

confidence: 99%

Functional gaze prediction in egocentric video

Naas

Jiang

Sigg

et al. 2020

Proceedings of the 18th International Conference on Advances in Mobile Computing &Amp; Multimedia

View full text Add to dashboard Cite

Streaming 360°videos to a head-mounted display (HMD) client is challenging due to their high network resource consumption and computational load. This is due to the use of gaze point prediction or image saliency features from the field of view (FoV) since, in real-time scenarios, FoV extraction is computationally demanding. We propose a functional gaze prediction system that addresses these issues by relying on a tiling scheme for gaze prediction. We condition gaze point prediction on virtual reality (VR) content and long short-term memory (LSTM)-encoded eye movement history. Further, we encode image flow and saliency maps of RGB images via VGG16, using a convolutional neural network (CNN). Future gaze points are then predicted using a novel sinusoidal encoding technique. In experiments, our tile-based approach outperforms state-of-the-art FoV-based schemes in terms of computational load and predicted gaze position. CCS CONCEPTS• Human-centered computing → Ubiquitous and mobile computing systems and tools; Virtual reality; Mixed / augmented reality; Mobile devices; • Computing methodologies → Probabilistic reasoning; Mixed / augmented reality.

show abstract

DGaze: CNN-Based Gaze Prediction in Dynamic Scenes

Cited by 60 publications

References 39 publications

Gazing at Social Interactions Between Foraging and Decision Theory

Gazing at Social Interactions Between Foraging and Decision Theory

On Gaze Deployment to Audio-Visual Cues of Social Interactions

Functional gaze prediction in egocentric video

Contact Info

Product

Resources

About