Toolbox and dataset for the development of saliency and scanpath models for omnidirectional/360° still images

Gutiérrez, Jesús; David, Erwan; Rai, Yashas; Callet, Patrick Le

doi:10.1016/j.image.2018.05.003

Cited by 72 publications

(39 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Four popular objective metrics in the saliency community are used, including Kullback-Leibler Divergence (KLD), Pearson's Correlation Coefficient (CC), Normalized Saliency Scanpath (NSS) and AUC-Judd [20]. The toolbox [21] is used to compute these scores. Table 2 reports the performance of the CMP-based saliency prediction for the test image P33.…”

Section: Resultsmentioning

confidence: 99%

Saliency Prediction for Omnidirectional Images Considering Optimization on Sphere Domain

Dedhia

Chiang

Char

2019

ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

There are several formats to describe the omnidirectional images. Among them, equirectangular projection (ERP), represented as 2D image, is the most widely used format. There exist many outstanding methods capable of well predicting the saliency maps for the conventional 2D images. But these works cannot be directly extended to predict the saliency map of the ERP image, since the content on ERP is not for direct display. Instead, the viewport image on demand is generated after converting the ERP image to the sphere domain, followed by rectilinear projection. In this paper, we propose a model to predict the saliency maps of the ERP images using existing saliency predictors for the 2D image. Some pre-processing and post-processing are used to manage the problem mentioned above. In particular, a smoothing based optimization is realized on the sphere domain. A public dataset of omnidirectional images is used to perform all the experiments and competitive results are achieved.

show abstract

Section: Resultsmentioning

confidence: 99%

Saliency Prediction for Omnidirectional Images Considering Optimization on Sphere Domain

Dedhia

Chiang

Char

2019

ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…The proposed FISEM model is evaluated by using the Salient360! [42] head only dataset. The dataset consists of 85 omnidirectional images and their corresponding ground truth saliency maps.…”

Section: Resultsmentioning

confidence: 99%

A Feature Integrated Saliency Estimation Model for Omnidirectional Immersive Images

et al. 2019

View full text Add to dashboard Cite

Omnidirectional, or 360°, cameras are able to capture the surrounding space, thus providing an immersive experience when the acquired data is viewed using head mounted displays. Such an immersive experience inherently generates an illusion of being in a virtual environment. The popularity of 360 • media has been growing in recent years. However, due to the large amount of data, processing and transmission pose several challenges. To this aim, efforts are being devoted to the identification of regions that can be used for compressing 360 • images while guaranteeing the immersive feeling. In this contribution, we present a saliency estimation model that considers the spherical properties of the images. The proposed approach first divides the 360 • image into multiple patches that replicate the positions (viewports) looked at by a subject while viewing a 360 • image using a head mounted display. Next, a set of low-level features able to depict various properties of an image scene is extracted from each patch. The extracted features are combined to estimate the 360 • saliency map. Finally, bias induced during image exploration and illumination variation is fine-tuned for estimating the final saliency map. The proposed method is evaluated using a benchmark 360 • image dataset and is compared with two baselines and eight state-of-the-art approaches for saliency estimation. The obtained results show that the proposed model outperforms existing saliency estimation models.2 of 15 approaches towards the saliency estimation rely on the detection of image components attracting human attention, i.e., color, intensity, and texture [6,7].Other approaches exploit Gestalt's psychological studies [8,9], according to which human perception focuses on figures more than on background elements. In [10] a Boolean map based saliency model (BMS) for 2D images is presented and an extended version of this approach, the extended Boolean map saliency approach (EBMS), is proposed in [11].Both BMS and EBMS do not consider the geometry-related features of an image and cannot directly be used for omnidirectional content since they do not address the problem of spherical projection of 360 • images that may cause artifacts. Fang et al. [12] adapt the traditional 2D saliency approach to 360 • images. Other methods adopt low-level features [12,13] or a combination of low and high-level features [14][15][16] for 360 • image saliency estimation.Recently, methods have been proposed to take into account the artifacts caused by the spherical projections. In [17], performances of three saliency estimation methods: Graph-based visual saliency, ensemble of deep networks (eDN), and the saliency attentive model (SAM) are compared. eDN exploits a six-multilayer structure to identify the salient regions through hyper-parameter optimisation. The ResNet architecture combined with pre-trained VGG-16 is used in the SAM model. Each model is tested using three types of 360 • image projection formats: Continuity, cube, and combined equirectangular image projection.Low...

show abstract

“…All these metrics were designed with traditional 2D images in mind. Gutiérrez et al [32] developed a toolbox that is specifically tailored to analyse ODIs. The results we present were obtained using these new tools.…”

Section: Performance Measuresmentioning

confidence: 99%

SalNet360: Saliency maps for omni-directional images with CNN

Monroy

Lutz

Chalasani

et al. 2018

Signal Processing: Image Communication

137

View full text Add to dashboard Cite

The prediction of Visual Attention data from any kind of media is of valuable use to content creators and used to efficiently drive encoding algorithms. With the current trend in the Virtual Reality (VR) field, adapting known techniques to this new kind of media is starting to gain momentum. In this paper, we present an architectural extension to any Convolutional Neural Network (CNN) to fine-tune traditional 2D saliency prediction to Omnidirectional Images (ODIs) in an end-to-end manner. We show that each step in the proposed pipeline works towards making the generated saliency map more accurate with respect to ground truth data.

show abstract

Toolbox and dataset for the development of saliency and scanpath models for omnidirectional/360° still images

Cited by 72 publications

References 22 publications

Saliency Prediction for Omnidirectional Images Considering Optimization on Sphere Domain

Saliency Prediction for Omnidirectional Images Considering Optimization on Sphere Domain

A Feature Integrated Saliency Estimation Model for Omnidirectional Immersive Images

SalNet360: Saliency maps for omni-directional images with CNN

Contact Info

Product

Resources

About