Deep Filter Banks for Texture Recognition, Description, and Segmentation

Abstract

Visual textures have played a key role in image understanding because they convey important semantics of images, and because texture representations that pool local image descriptors in an orderless manner have had a tremendous impact in diverse applications. In this paper we make several contributions to texture understanding. First, instead of focusing on texture instance and material category recognition, we propose a human-interpretable vocabulary of texture attributes to describe common texture patterns, complemented by a new describable texture dataset for benchmarking. Second, we look at the problem of recognizing materials and texture attributes in realistic imaging conditions, including when textures appear in clutter, developing corresponding benchmarks on top of the recently proposed OpenSurfaces dataset. Third, we revisit classic texture represenations, including bag-of-visual-words and the Fisher vectors, in the context of deep learning and show that these have excellent efficiency and generalization properties if the convolutional layers of a deep model are used as filter banks. We obtain in this manner state-of-the-art performance in numerous datasets well beyond textures, an efficient method to apply deep features to image regions, as well as benefit in transferring features from one domain to another.

GEO-CEOS stage 4 validation of the Satellite Image Automatic Mapper lightweight computer program for ESA Earth observation level 2 product generation - Part 1: Theory

“…In 1D image analysis, a 1D streamline of vector data, either spatial context-sensitive (e.g., window-based or image object-based like in OBIA approaches) or spatial context-insensitive (pixel-based), is processed insensitive to changes in the order of presentation of the input sequence. In practice 1D image analysis is invariant to permutations, such as in orderless pooling encoders (Cimpoi et al, 2014). When vector data are spatial context-sensitive then 1D image analysis ignores spatial topological information.…”

“…When it is input to a traditional inductive data learning classifier, this 1D vector data stream is what the inductive classifier actually sees when watching the (2D) image at left. Undoubtedly, computers are more successful than humans in 1D image analysis, invariant to permutations in the input vector data sequence, such as in orderless pooling encoders (Cimpoi et al, 2014). Nonetheless, humans are still far more successful than computers in 2D image analysis, synonym of spatial topology-preserving (retinotopic) image analysis (Tsotsos, 1990), sensitive to permutations in the input vector data sequence, such as in order-sensitive pooling encoders (Cimpoi et al, 2014).…”

“…Alternative to 1D image analysis, 2D image analysis relies on a sparse (distributed) 2D array (2D regular grid) of local spatial filters, suitable for spatial topology-preserving (retinotopic) feature mapping (DiCarlo, 2017; Fritzke, 1997a; Martinetz, Berkovich, & Schulten, 1994; Tsotsos, 1990), sensitive to permutations in the input vector data sequence, such as in order-sensitive pooling encoders (Cimpoi et al, 2014), see Figure 16. The human brain’s organizing principle is topology-preserving feature mapping (Feldman, 2013).…”

“…On the one hand, it agrees with a minor portion of the RS literature where supervised data learning classification of EO image datasets at continental or global spatial extent into binary LC class vegetation/non-vegetation is considered very challenging (Gutman et al, 2004). On the other hand, it is at odd with the RS mainstream, where the semantic information gap from sub-symbolic EO data to multi-class LC taxonomies is typically filled in one step, implemented as a supervised data learning classifier (Bishop, 1995; Cherkassky & Mulier, 1998), e.g., a support vector machine, random forest or deep convolutional neural network (DCNN) (Cimpoi, Maji, Kokkinos, & Vedaldi, 2014), which is equivalent to an unstructured black box (Marcus, 2018), inherently semiautomatic and site specific (Liang, 2004) and whose opacity contradicts the well-known engineering principles of modularity, regularity and hierarchy typical of scalable systems (Lipson, 2007).

10.1080/23312041.2018.1467357-F0003Figure 3.The fully nested 3-level 8-class FAO Land Cover Classification System (LCCS) Dichotomous Phase (DP) taxonomy consists of a sorted set of 3 dichotomous layers: (i) vegetation versus non-vegetation, (ii) terrestrial versus aquatic, and (iii) managed versus natural or semi-natural.

…”

“…One of the most widely used VQ heuristics in RS and computer vision (CV) applications is the k -means VQ algorithm (Elkan, 2003; Lee et al, 1997; Linde et al, 1980; Lloyd, 1982), capable of convex Voronoi tessellation of a multi-variate data space (Cherkassky & Mulier, 1998; Fritzke, 1997a). For example, in a bag-of-words model applied to CV tasks, a numeric color space is typically discretized into a categorical color variable (codebook of codewords) by an inductive VQ algorithm, such as k -means; next, the categorical color variable is simplified by a 1st-order histogram representation, which disregards word grammar, semantics and even word-order, but keeps multiplicity; finally, the frequency of each color codeword is used as a feature for training a supervised data learning classifier (Cimpoi et al, 2014). Unlike the k -means VQ algorithm where the system’s free-parameter k is user-defined based on heuristics and the VQ error is estimated from the unlabeled dataset at hand, a user can fix the target VQ error value, so that it is the free-parameter k to be dynamically learned from the finite unlabeled dataset at hand by an inductive VQ algorithm (Patané & Russo, 2001, 2002), such as ISODATA (Memarsadeghi, Mount, Netanyahu, & Le Moigne, 2007).…”