Integrating a Model for Visual Attention into a System for Natural Language Parsing

Abstract

We present a system for integrating knowledge about complex visual scenes into the process of natural language comprehension. The implemented system is able to choose a scene of reference for a natural language sentence from a large set of scene descriptions. This scene is then used to influence the analysis of a sentence generated by a broad coverage language parser. In addition, objects and actions referred to by the sentence are visualized by a saliency map which is derived from the bi-directional influence of top down and bottomup information on a model of visual attention highlighting the regions with the highest probability of attracting the attention of an observer.