Natural-language retrieval of images based on descriptive captions

Abstract

We describe a prototype intelligent information retrieval system that uses natural-language understanding to efficiently locate captioned data. Multimedia data generally require captions to explain their features and significance. Such descriptive captions often rely on long nominal compounds (strings of consecutive nouns) which create problems of disambiguating word sense. In our system, captions and user queries are parsed and interpreted to produce a logical form, using a detailed theory of the meaning of nominal compounds. A fine-grain match can then compare the logical form of the query to the logical forms for each caption. To improve system efficiency, we first perform a coarse-grain match with index files, using nouns and verbs extracted from the query. Our experiments with randomly selected queries and captions from an existing image library show an increase of 30% in precision and 50% in recall over the keyphrase approach currently used. Our processing times have a median of seven seconds as compared to eight minutes for the existing system, and our system is much easier to use.

NLP for indexing and retrieval of captioned photographs

“…Researchers in information retrieval argue that detailed linguistic analysis is usually unnecessary to improve accuracy for text indexing and retrieval; however, in the case of captioned photographs, natural language processing (NLP) techniques have proved to be particularly effective for the very same tasks (Rose et al, 2000;Guglielmo and Rowe, 1996).…”

“…For SOCIS, we have attempted a pragmatic, corpus-based approach, where the set of primitives emerge from the data. MARIE (Guglielmo and Rowe, 1996) is a system that uses a domain lexicon and a type hierarchy to represent both queries and captions in a logical form and then matches these representations instead of mere keywords; the logical forms are case grammar constructs structured in a slot-assertion notation.…”

Natural Language Query vs. Keyword Search: Effects of Task Complexity on Search Performance, Participant Perceptions, and Preferences

“…In order to provide better search experience for average participants, researchers have applied natural langauge processing (NLP) technology to building information retrieval systems [2][3][4][5][6]. However, usability studies concerning seeking and interacting with information have focused on keyword search (KW) rather than natural language [7][8][9][10][11][12][13][14][15][16].…”

Automatic caption localization for photographs on World Wide Web pages

“…It assumed captions were already extracted for their pictures, an assumption not applicable on World Wide Web. Our own MARIE Project in the MARIE-1 system, Guglielmo and Rowe (1996), and MARIE-2 system, Rowe (1996), explored similar issues in a large photographic library at the NAWC-WD Navy facility in China Lake, California USA. Rowe and Frew (1997) used knowledge of a technical vocabulary and specialized syntax to aid image processing in categorizing regions in photographs, with the goal of retrieval by content.…”