Visual Material Traits: Recognizing Per-Pixel Material Context

Gabriel Schwartz, Ko Nishino

Abstract: Information describing the materials that make up scene constituents provides invaluable context that can lead to a better understanding of images. We would like to obtain such material information at every pixel, in arbitrary images, regardless of the objects involved. In this paper, we introduce visual material traits to achieve this. Material traits, such as "shiny," or "woven," encode the appearance of characteristic material properties. We learn convolution kernels in an unsupervised setting to recognize complex material trait appearances at each pixel. Unlike previous methods, our framework explicitly avoids influence from object-specific information. We may, therefore, accurately recognize material traits regardless of the object exhibiting them. Our results show that material traits are discriminative and can be accurately recognized. We demonstrate the use of material traits in material recognition and image segmentation. To our knowledge, this is the first method to extract and use such per-pixel material information.