Humans can often estimate tactile properties of objects from vision alone. For example, during online shopping, we can often infer material properties of clothing from images and judge how the material would feel against our skin. What visual information is important for tactile perception? Previous studies in material perception have focused on measuring surface appearance, such as gloss and roughness, and using verbal reports of material attributes and categories. However, in real life, predicting tactile properties of an object might not require accurate verbal descriptions of its surface attributes or categories. In this paper, we use tactile perception as ground truth to measure visual material perception. Using fabrics as our stimuli, we measure how observers match what they see (photographs of fabric samples) with what they feel (physical fabric samples). The data shows that color has a significant main effect in that removing color significantly reduces accuracy, especially when the images contain 3-D folds. We also find that images of draped fabrics, which revealed 3-D shape information, achieved better matching accuracy than images with flattened fabrics. The data shows a strong interaction between color and folding conditions on matching accuracy, suggesting that, in 3-D folding conditions, the visual system takes advantage of chromatic gradients to infer tactile properties but not in flattened conditions. Together, using a visual-tactile matching task, we show that humans use folding and color information in matching the visual and tactile properties of fabrics.

Figure 1: Dynamic information is important for perception of mechanical properties of cloth. Using this information, however, can be challenging because both the intrinsic mechanical properties and the external forces contribute to the apparent motion of fabrics. For example, lighter fabrics under weak wind can look similar to heavier fabrics under strong wind (the third image in the first row and the fourth image in second row). Can humans discount the external wind and have an invariant inference of the intrinsic mechanical properties of the cloth? If so, what visual cues allow them to achieve such constancy? We create animated videos where we we vary both the intrinsic mechanical properties (bending stiffness and mass) as well as the wind force, and measure the perception of mechanical properties of fabrics.

Humans can visually estimate the mechanical properties of deformable objects (e.g. cloth stiffness). While much of the recent work on material perception has focused on static image cues (e.g., textures and shape), little is known whether humans can integrate information over time to make a judgment. Here, we investigate the effect of spatiotemporal information across multiple frames (multi-frame motion) on estimating the bending stiffness of cloth. Using high-fidelity cloth animations, we first examined how the perceived bending stiffness changed as a function of the physical bending stiffness defined in the simulation model. Using maximum likelihood difference scaling methods (MLDS) we found that the perceived stiffness and the physical bending stiffness were highly correlated. A second experiment in which we scrambled the frame sequences diminished this correlation. This suggests that multi-frame motion plays an important role. To provide further evidence for this finding, we extracted dense motion trajectories from the videos across 15 consecutive frames and used the trajectory descriptors to train a machine-learning model with the measured perceptual scales. The model can predict human perceptual scales in new videos with varied winds, optical properties of cloth, and scene setups. When the correct multi-frame was removed (using either scrambled videos or 2-frame optical flow to train the model), the predictions significantly worsened. Our findings demonstrate that multi-frame motion information is important for both humans and machines to estimate the mechanical properties. In addition, we show that dense motion trajectories are effective features to build a successful automatic cloth estimation system.

Humans can visually estimate the mechanical properties of deformable objects (e.g., cloth stiffness). While much of the recent work on material perception has focused on static image cues (e.g., textures and shape), little is known about whether humans can integrate information over time to make a judgment. Here we investigated the effect of spatiotemporal information across multiple frames (multiframe motion) on estimating the bending stiffness of cloth. Using high-fidelity cloth animations, we first examined how the perceived bending stiffness changed as a function of the physical bending stiffness defined in the simulation model. Using maximum-likelihood difference-scaling methods, we found that the perceived stiffness and physical bending stiffness were highly correlated. A second experiment in which we scrambled the frame sequences diminished this correlation. This suggests that multiframe motion plays an important role. To provide further evidence for this finding, we extracted dense motion trajectories from the videos across 15 consecutive frames and used the trajectory descriptors to train a machine-learning model with the measured perceptual scales. The model can predict human perceptual scales in new videos with varied winds, optical properties of cloth, and scene setups. When the correct multiframe was removed (using either scrambled videos or two-frame optical flow to train the model), the predictions significantly worsened. Our findings demonstrate that multiframe motion information is important for both humans and machines to estimate the mechanical properties. In addition, we show that dense motion trajectories are effective features to build a successful automatic cloth-estimation system.

Cloth is a common material, and humans can visually estimate its mechanical properties by observing how it deforms under external forces. Here, we ask whether and how dynamic deformation can affect the perception of mechanical properties of cloth. In Experiment 1, we find that both intrinsic mechanical properties and optical properties affect stiffness perception when the stimuli are presented as images. By contrast, in videos, humans can partially discount the effect of optical appearances and exhibit higher sensitivity to stiffness. We further identified an idiosyncratic deformation pattern (i.e., movement uniformity) to differentiate stiffness, which can be reliably measured by six optical flow features. In Experiment 2, we isolate the deformation by creating dynamic dot stimuli from the 3-D mesh of the cloth. We directly alter the movement pattern by manipulating the uniformity of the displacement vectors on the dot stimuli and show that changing the pattern of dynamic deformation alone can alter the perceived stiffness of cloth in a variety of scene setups. Furthermore, by analyzing optical flow fields extracted from the manipulated dynamic dot stimuli, we confirmed the same six optical flow features can be diagnostic of the degree of stiffness of moving cloth across different scenes. Overall, our study demonstrates that manipulating patterns of dynamic deformation alone can elicit the impression of cloth with varying stiffness, suggesting that the human visual system might rely on the idiosyncratic pattern of dynamic deformation for estimating stiffness.