Scene Modeling and Image Mining With a Visual Grammar

Abstract

Automatic content extraction, classification and content-based retrieval are highly desired goals in intelligent remote sensing databases. Pixel level processing has been the common choice for both academic and commercial systems. We extend the modeling of remotely sensed imagery to three levels: Pixel level, region level and scene level. Pixel level features are generated using unsupervised clustering of spectral values, texture features and ancillary data like digital elevation models. Region level features include shape information and statistics of pixel level feature values. Scene level features include statistics and spatial relationships of regions. This chapter describes our work on developing a probabilistic visual grammar to reduce the gap between low-level features and high-level user semantics, and to support complex query scenarios that consist of many regions with different feature characteristics. The visual grammar includes automatic identification of region prototypes and modeling of their spatial relationships. The system learns the prototype regions in an image collection using unsupervised clustering. Spatial relationships are represented by fuzzy membership functions. The system automatically selects significant relationships from training data and builds visual grammar models which can also be updated using user relevance feedback. A Bayesian framework is used to automatically classify scenes based on these models. We demonstrate our system with query scenarios that cannot be expressed by traditional region or scene level approaches but where the visual grammar provides accurate classifications and effective retrieval.

Spatial Techniques for Image Classification

“…In previous work [25], we used an automatic segmentation algorithm that breaks an image into many small regions and merges them by minimizing an energy functional that trades off the similarity of regions against the length of their shared boundaries. We have also recently experimented with several segmentation algorithms from the computer vision literature.…”

Modeling of Remote Sensing Image Content Using Attributed Relational Graphs

“…In previous work [8], we used an automatic segmentation algorithm based on energy minimization, and used k-means and Gaussian mixture-based clustering algorithms to group and label the resulting regions according to their features. Our newer experiments showed that some popular density-based and graphtheoretic segmentation algorithms were not successful on our data sets because of the large amount of data and the detailed structure in multi-spectral images.…”

“…These computations can be significantly simplified by applying a coarse-to-fine search to find region pairs that have a potential overlap or are very close to each other. In previous work [8,9], we used brute force comparisons of region pairs within smaller tiles obtained by dividing the original scene into manageable sized images. However, regions that occupy multiple tiles may not be handled correctly after that division.…”

“…Details of the membership functions are not included here due to space restrictions but more information can be found in [8].…”

“…In previous work [8], we modeled higher-order relationships (of region groups) by decomposing them into k 2 second-order relationships (of region pairs) combined using the fuzzy "min" operator that corresponds to the Boolean "and" operator. In this work, we model higher-order relationships using attributed relational graph (ARG) structures.…”