Visual Scanpath Prediction Using IOR-ROI Recurrent Mixture Density Network

Sun, Wanjie; Chen, Zhenzhong; Wu, Feng

doi:10.1109/tpami.2019.2956930

Cited by 44 publications

(41 citation statements)

References 62 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The IOR-ROI-LSTM model (Sun et al 2019) uses deep learning models to encode an image into deep features and predicted semantic masks. This data is then feed into a recurrent neural network architecture which uses channel-wise attention to inhibit certain image features and predict the next region of interest via a mixture of Gaussians.…”

Section: Ior-roi-lstmmentioning

confidence: 99%

“…Several existing scanpath models draw multiple candidates from some distribution, compute a gain for each candidate and then select the candidate with the highest gain, e.g., Boccignone and Ferraro (2004), Le Meur and Liu (2015) and Sun et al (2019). This pattern, which we will call best-of-k sampling in the following, is convenient for sampling fixations but it makes it nontrivial to compute the full conditional model distribution p(f i | f 0 , .…”

Section: Best-of-k Samplingmentioning

confidence: 99%

See 1 more Smart Citation

State-of-the-Art in Human Scanpath Prediction

Kümmerer,

Bethge

2021

Preprint

View full text Add to dashboard Cite

The last years have seen a surge in models predicting the scanpaths of fixations made by humans when viewing images. However, the field is lacking a principled comparison of those models with respect to their predictive power. In the past, models have usually been evaluated based on comparing human scanpaths to scanpaths generated from the model. Here, instead we evaluate models based on how well they predict each fixation in a scanpath given the previous scanpath history. This makes model evaluation closely aligned with the biological processes thought to underly scanpath generation and allows to apply established saliency metrics like AUC and NSS in an intuitive and interpretable way. We evaluate many existing models of scanpath prediction on the datasets MIT1003, MIT300, CAT2000 train and CAT200 test, for the first time giving a detailed picture of the current state of the art of human scanpath prediction. We also show that the discussed method of model benchmarking allows for more detailed analyses leading to interesting insights about where and when models fail to predict human behaviour. The MIT/Tuebingen Saliency Benchmark will implement the evaluation of scanpath models as detailed here, allowing researchers to score their models on the established benchmark datasets MIT300 and CAT2000.

show abstract

Section: Ior-roi-lstmmentioning

confidence: 99%

Section: Best-of-k Samplingmentioning

confidence: 99%

State-of-the-Art in Human Scanpath Prediction

Kümmerer,

Bethge

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…It is now widely admitted that when an image is viewed, the HVS gazes on salient details, which translates into eye fixations [14]. In our case, these regions are considered as our viewports and are detected using the visual scan-path model proposed in [15]. This model provides trajectories including the order and duration of fixations.…”

Section: Pre-processingmentioning

confidence: 99%

Perceptually-Weighted Cnn For 360-Degree Image Quality Assessment Using Visual Scan-Path And Jnd

Sendjasni¹,

Larabi²,

Cheikh³

2021

2021 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

Image quality assessment of immersive content and more specifically 360-degree one is still in its infancy. There are many challenges regarding sphere vs. projected representation, human visual system (HVS) properties in a 360-degree environment, etc. In this paper, we propose the use of CNNs to design a no reference model to predict visual quality of 360-degree images. Instead of feeding the CNN with ERPs, visually important viewports are extracted based on visual scan-path prediction and given to a multi-channel CNN using DenseNet-121. Moreover, information about visual fixations and just noticeable difference are used to account for the HVS properties and make the network closer to human judgment. The scan-path is also used to create multiple instances of the database so as to perform a robust generalization analysis and compensate for the lack of databases.

show abstract

“…At present, there exist multiple ways to model cross-data relationships, e.g., recurrent neural network (RNN) [64], long short-term memory (LSTM) [65], [66], and gate recurrent unit (GRU) [67]. Although these existing tools can sense subtle changes over time, they are clearly not suitable in our case because these tools require their input data to be spatially well aligned.…”

Section: Multigranularity Perceptionmentioning

confidence: 99%

Weakly Supervised Visual-Auditory Saliency Detection with Multigranularity Perception

Wang¹,

Chen²,

Fan³

et al. 2021

Preprint

View full text Add to dashboard Cite

Thanks to the rapid advances in deep learning techniques and the wide availability of large-scale training sets, the performance of video saliency detection models has been improving steadily and significantly. However, deep learning-based visualaudio fixation prediction is still in its infancy. At present, only a few visual-audio sequences have been furnished, with real fixations being recorded in real visual-audio environments. Hence, it would be neither efficient nor necessary to recollect real fixations under the same visual-audio circumstances. To address this problem, this paper promotes a novel approach in a weakly supervised manner to alleviate the demand of large-scale training sets for visual-audio model training. By using only the video category tags, we propose the selective class activation mapping (SCAM) and its upgrade (SCAM+). In the spatial-temporal-audio circumstance, the former follows a coarse-to-fine strategy to select the most discriminative regions, and these regions are usually capable of exhibiting high consistency with the real human-eye fixations. The latter equips the SCAM with an additional multi-granularity perception mechanism, making the whole process more consistent with that of the real human visual system. Moreover, we distill knowledge from these regions to obtain complete new spatial-temporal-audio (STA) fixation prediction (FP) networks, enabling broad applications in cases where video tags are not available. Without resorting to any real human-eye fixation, the performances of these STA FP networks are comparable to those of fully supervised networks. The code and results are publicly available at https://github.com/guotaowang/STANet.

show abstract

Visual Scanpath Prediction Using IOR-ROI Recurrent Mixture Density Network

Cited by 44 publications

References 62 publications

State-of-the-Art in Human Scanpath Prediction

State-of-the-Art in Human Scanpath Prediction

Perceptually-Weighted Cnn For 360-Degree Image Quality Assessment Using Visual Scan-Path And Jnd

Weakly Supervised Visual-Auditory Saliency Detection with Multigranularity Perception

Contact Info

Product

Resources

About