Saliency-based identification and recognition of pointed-at objects

Schauerte, Boris; Richarz, Jan; Fink, Gernot A.

doi:10.1109/iros.2010.5649430

Cited by 27 publications

(34 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Schauerte et al [7] determines the pointing direction of the human and uses that to find the object of interest in the saliency map by locally searching for the most salient point.…”

Section: Existing Modelsmentioning

confidence: 99%

Multimodal joint visual attention model for natural human-robot interaction in domestic environments

Domhof

Chandarr

Rudinac

et al. 2015

2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

Due to population ageing, the cost of health care will raise in the coming years. One way to help humans, and especially elderly people, is the introduction of domestic robots that can assist people in daily life such that they are less dependent on home care. Joint visual attention models can be used for natural robot-human interaction. Joint visual attention is that two humans or a robot and a human have a shared attention to the same object. This can be accomplished by pointing, eye-gaze or by using speech. The goal of this thesis is to develop a non verbal joint visual attention model for object detection that integrates gestures, gaze, saliency and depth. The question that will be answered in this report is: how can the information from gestures, gaze, saliency and depth be integrated in the most efficient way to determine the object of interest?Existing joint visual attention models only work when the human is in front of the robot, so that the human is in view of the camera. Our model should be more flexible than existing models, so it needs to work in different configurations of human, robot and object. Furthermore, the joint visual attention model should be able to determine the object of interest when the pointing direction or the gaze location is not available.The saliency algorithm of Itti et al. [1] has been used to create a bottom up saliency map. The second bottom-up cue, depth, is determined by means of segmenting the environment to extract the objects. Apart from the bottom-up cues, top-down cues can be used as well. The pointing finger is identified and based on the eigenvalues and eigenvectors of the finger the pointing direction will be retrieved. A pointing map is created by means of the angle between the 3D pointing direction vector and the 3D vector from the pointing finger to the object. A hybrid model, which computes a gaze map, has been developed that switches depending on textureness of the object between texture based approach and color based approach.Depending on the configuration of the human, robot and object, three or four maps are available to determine the object of interest. In some configurations, the pointing map or gaze map is not available. In that case the combined saliency map is obtained by point wise multiplication of these three maps. If all four maps are at our disposal, all maps are added and multiplied by the pointing mask.When the human and robot are opposite of each other and pointing, bottom up saliency and depth are combined, 93.3% of the objects are detected correctly. If the human is standing next to the robot, the gaze map, bottom up saliency map and depth map are combined, then the detection rate is 67.8%. If robot, human and object are standing in a triangular shape, the detection rate is equal to 96.3%.The main contribution is that the joint visual attention model is able to detect objects of interest in different configurations of human, robot and object and it also works when one of the four cues is not available. Furthermore, a hybrid model has ...

show abstract

“…Schauerte et al [7] determines the pointing direction of the human and uses that to find the object of interest in the saliency map by locally searching for the most salient point.…”

Section: Existing Modelsmentioning

confidence: 99%

Multimodal joint visual attention model for natural human-robot interaction in domestic environments

Domhof

Chandarr

Rudinac

et al. 2015

2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

show abstract

“…Since reviewing them is beyond the scope of this paper, we recommend reading the survey of computational visual attention in [11]. In this contribution, we apply a visual saliency model that is based on spectral whitening of the image signal (see [3], [18]- [20]), which exploits that the elimination of a signals' magnitude components accentuates narrow spatial events [21].…”

Section: Related Workmentioning

confidence: 99%

“…Accordingly, a common problem of computational visual attention models is to determine the image region around the selected focus of attention that approximates the extent of a (proto-)object at that location (see [22]). To this end, various conventional segmentation methods are applied (see, e.g., [9], [18], [19], [22]- [25]), e.g. region growing [9] and maximally stable extremal regions [3], [18], and even feedback connections in the saliency computation hierarchy have been introduced [22].…”

Section: Related Workmentioning

confidence: 99%

“…To this end, various conventional segmentation methods are applied (see, e.g., [9], [18], [19], [22]- [25]), e.g. region growing [9] and maximally stable extremal regions [3], [18], and even feedback connections in the saliency computation hierarchy have been introduced [22]. We introduce a novel method that analyzes the isophote curvature (see [26]) of the saliency map to approximate proto-object regions.…”

Section: Related Workmentioning

confidence: 99%

“…To extract all salient proto-object regions that attract the attention, we apply a location-based inhibition of return (see [40]) mechanism on the saliency map (see, e.g., [5], [8], [15], [18]). To this end, we use the accumulator to select the most salient proto-object region and inhibit all pixels within the estimated outline by setting their saliency to zero.…”

Section: Systemmentioning

confidence: 99%

See 2 more Smart Citations

Multimodal saliency-based attention for object-based scene analysis

Schauerte

Kühn

Kroschel

et al. 2011

2011 IEEE/RSJ International Conference on Intelligent Robots and Systems

View full text Add to dashboard Cite

Abstract-Multimodal attention is a key requirement for humanoid robots in order to navigate in complex environments and act as social, cognitive human partners. To this end, robots have to incorporate attention mechanisms that focus the processing on the potentially most relevant stimuli while controlling the sensor orientation to improve the perception of these stimuli. In this paper, we present our implementation of audio-visual saliency-based attention that we integrated in a system for knowledge-driven audio-visual scene analysis and object-based world modeling. For this purpose, we introduce a novel isophote-based method for proto-object segmentation of saliency maps, a surprise-based auditory saliency definition, and a parametric 3-D model for multimodal saliency fusion. The applicability of the proposed system is demonstrated in a series of experiments.Index Terms-audio-visual saliency, auditory surprise, isophote-based visual proto-objects, parametric 3-D saliency model, object-based inhibition of return, multimodal attention, scene exploration, hierarchical object analysis, overt attention, active perception

show abstract

Applications of Saliency Models

Mancaş¹,

Meur²

2016

From Human Attention to Computational Attention

View full text Add to dashboard Cite

International audienceThis chapter proposes a taxonomy to classify the real-life applications which can benefit from the use of attention models. There are numerous applications and we try here to remain as exhaustive as possible to provide a picture of all the applications of saliency models, but also to detect where future developments might be of interest. The applications are grouped into three categories. The first one uses the detection of the most important regions in an image and contains applications such as video surveillance, audio surveillance, defect detection, pathology detection, expressive and social gestures, computer graphics and quality metrics. The second category uses saliency maps to detect the regions which are the less interesting in an image. Here one can found applications like texture metrics, compression, retargeting, summarization, watermarking and attention-based ad insertion. Finally, a third category uses the most interesting areas in an image with further processing like comparisons between those areas. In this category one can find image registration and landmarks, object recognition, action guidance in robotics or avatars, web sites optimization, images memorability, best viewpoint, symmetries and automatic focus on images

show abstract

Saliency-based identification and recognition of pointed-at objects

Cited by 27 publications

References 43 publications

Multimodal joint visual attention model for natural human-robot interaction in domestic environments

Multimodal joint visual attention model for natural human-robot interaction in domestic environments

Multimodal saliency-based attention for object-based scene analysis

Applications of Saliency Models

Contact Info

Product

Resources

About