Inferring Attention Shift Ranks of Objects for Image Saliency

Siris, Avishek; Jiao, Jianbo; Tam, Gary K. L.; Xie, Xianghua; Lau, Rynson W. H.

doi:10.1109/cvpr42600.2020.01215

Cited by 38 publications

(66 citation statements)

References 66 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The current popular saliency detection datasets do not contain depth information. It could therefore be concluded that the study of saliency detection algorithms in relation to RGB-D content is seen as a current challenge for this particular area [21] [23] [13] and the dataset being proposed in this paper aims to explore this further.…”

Section: A Saliency Detection Datasetsmentioning

confidence: 99%

COTS: A Multipurpose RGB-D Dataset for Saliency and Image Manipulation Applications

et al. 2021

View full text Add to dashboard Cite

Many modern computer vision systems include several modules that perform different processing operations packaged as a single pipeline architecture. This generally introduces a challenge in the evaluation process since most datasets provide evaluation data for just one of the operations. In this paper, we present an RGB-D dataset that was designed from first principles to cater for applications that involve salient object detection, segmentation, inpainting and blending techniques. This addresses a gap in the evaluation of image inpainting and blending applications that generally rely on subjective evaluation due to the lack of availability of comparative data. A set of experiments were carried out to demonstrate how the COTS dataset can be used to evaluate these different applications. This dataset includes a variety of scenes, where each scene is captured multiple times, each time adding a new object to the previous scene. This allows for a comparative analysis at pixel level in image inpainting and blending applications. Moreover, all objects were manually labelled in order to offer the possibility of salient object detection even in scenes that contain multiple objects. An online test with 1267 participants was also carried out, and this dataset also includes the click coordinates of users' selection for every image, introducing a user interaction dimension in the same RGB-D dataset. This dataset was also validated using state of the art techniques, obtaining an F β of 0.957 in salient object detection and a mean (Intersection over Union) IoU of 0.942 in Segmentation. Results demonstrate that the COTS dataset introduces novel possibilities for the evaluation of computer vision applications.

show abstract

Section: A Saliency Detection Datasetsmentioning

confidence: 99%

COTS: A Multipurpose RGB-D Dataset for Saliency and Image Manipulation Applications

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Salient Instance Detection (SID) goes further from SOD as it aims to differentiate individual salient instances. This instance-level saliency information can benefit vision tasks that require fine-grained scene understanding, e.g., object rank [41], image captioning [21], image editing [59] and semantic segmentation [43]. However, existing SID methods [11,25,62] still rely on large-scale annotated ground truth masks in order to learn how to segment salient instances with their boundaries delineated.…”

Section: Introductionmentioning

confidence: 99%

Learning to Detect Instance-level Salient Objects Using Complementary Image Labels

Tian¹,

Xu²,

Yang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Existing salient instance detection (SID) methods typically learn from pixel-level annotated datasets. In this paper, we present the first weakly-supervised approach to the SID problem. Although weak supervision has been considered in general saliency detection, it is mainly based on using class labels for object localization. However, it is non-trivial to use only class labels to learn instance-aware saliency information, as salient instances with high semantic affinities may not be easily separated by the labels. As the subitizing information provides an instant judgement on the number of salient items, it is naturally related to detecting salient instances and may help separate instances of the same class while grouping different parts of the same instance. Inspired by this observation, we propose to use class and subitizing labels as weak supervision for the SID problem. We propose a novel weakly-supervised network with three branches: a Saliency Detection Branch leveraging class consistency information to locate candidate objects; a Boundary Detection Branch exploiting class discrepancy information to delineate object boundaries; and a Centroid Detection Branch using subitizing information to detect salient instance centroids. This complementary information is then fused to produce a salient instance map. To facilitate the

show abstract

“…E-mail: liunian228@gmail.com, ling.shao@inceptioniai.org • L. Li, W. Zhao, and J. Han are with School of Automation, Northwestern Polytechnical University, Xi'an, China, E-mail: {longli.nwpu,wangbo.zhao96,junweihan2010}@gmail.com • Junwei Han is the corresponding author. (d) RSDNet [12] (e) ASSR [13] (f) Ours In [12], the authors predicted pixel-wise saliency ranking, without actually differentiating object instances. (e): In [13], saliency ranking is inferred based on attention shift, and only less than five objects are considered.…”

Section: Introductionmentioning

confidence: 99%

“…(d) RSDNet [12] (e) ASSR [13] (f) Ours In [12], the authors predicted pixel-wise saliency ranking, without actually differentiating object instances. (e): In [13], saliency ranking is inferred based on attention shift, and only less than five objects are considered.…”

Section: Introductionmentioning

confidence: 99%

“…They have to use GT instance segmentation maps to obtain object-level relative saliency ranking results, which is infeasible for practical usage. In [13], Siris et al proposed to predict saliency ranking by inferring human attention shift, which describes how humans sequentially select and shift attention from one object to another. However, the degree of saliency of an object mainly depends on the duration of gaze instead of the sequential order in which objects are focused.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Instance-Level Relative Saliency Ranking with Graph Reasoning

Liu¹,

Li²,

Zhao³

et al. 2021

Preprint

View full text Add to dashboard Cite

Conventional salient object detection models cannot differentiate the importance of different salient objects. Recently, two works have been proposed to detect saliency ranking by assigning different degrees of saliency to different objects. However, one of these models cannot differentiate object instances and the other focuses more on sequential attention shift order inference. In this paper, we investigate a practical problem setting that requires simultaneously segment salient instances and infer their relative saliency rank order. We present a novel unified model as the first end-to-end solution, where an improved Mask R-CNN is first used to segment salient instances and a saliency ranking branch is then added to infer the relative saliency. For relative saliency ranking, we build a new graph reasoning module by combining four graphs to incorporate the instance interaction relation, local contrast, global contrast, and a high-level semantic prior, respectively. A novel loss function is also proposed to effectively train the saliency ranking branch. Besides, a new dataset and an evaluation metric are proposed for this task, aiming at pushing forward this field of research. Finally, experimental results demonstrate that our proposed model is more effective than previous methods. We also show an example of its practical usage on adaptive image retargeting.

show abstract

Inferring Attention Shift Ranks of Objects for Image Saliency

Cited by 38 publications

References 66 publications

COTS: A Multipurpose RGB-D Dataset for Saliency and Image Manipulation Applications

COTS: A Multipurpose RGB-D Dataset for Saliency and Image Manipulation Applications

Learning to Detect Instance-level Salient Objects Using Complementary Image Labels

Instance-Level Relative Saliency Ranking with Graph Reasoning

Contact Info

Product

Resources

About