Material recognition in the wild with the Materials in Context Database

Abstract

Recognizing materials in real-world images is a challenging task. Real-world materials have rich surface texture, geometry, lighting conditions, and clutter, which combine to make the problem particularly difficult. In this paper, we introduce a new, large-scale, open dataset of materials in the wild, the Materials in Context Database (MINC), and combine this dataset with deep learning to achieve material recognition and segmentation of images in the wild.MINC is an order of magnitude larger than previous material databases, while being more diverse and well-sampled across its 23 categories. Using MINC, we train convolutional neural networks (CNNs) for two tasks: classifying materials from patches, and simultaneous material recognition and segmentation in full images. For patch-based classification on MINC we found that the best performing CNN architectures can achieve 85.2% mean class accuracy. We convert these trained CNN classifiers into an efficient fully convolutional framework combined with a fully connected conditional random field (CRF) to predict the material at every pixel in an image, achieving 73.1% mean class accuracy. Our experiments demonstrate that having a large, well-sampled dataset such as MINC is crucial for real-world material recognition and segmentation.

On the appearance of translucent edges

“…Instead, data-driven, phenomenological approaches are more likely to be fruitful. Such approaches might benefit from the recent availability of large-scale material databases [3,4].…”

Crowdsourcing in Computer Vision

“…Intrinsic Images in the Wild [Bell et al, 2014] follows up by annotating millions of crowdsourced pairwise comparisons of material properties. Materials in Context Database [Bell et al, 2015] uses a three-stage Mechanical Turk pipeline to annotate three million material samples, significantly scaling up over the previous Flickr Material Dataset [Sharan et al, 2009] benchmark. These datasets enable research into deeper pixel-level image understanding.…”

Fine-Grained Material Classification Using Micro-geometry and Reflectance

“…But with the advent of the Internet and crowd-source annotation, collection and annotation of large databases of materials and textures was made feasible, which in turn brought renewed attention to the problem of material classification [8][9][10][11][12]. Currently the trend is to use materials captured in unconstrained conditions, also referred to as "in-the-wild", and mainly in the large scale [10,11,9,8]. In this paper, we take a different direction which has, to the best of our knowledge, has received limited attention.…”