Perception of soft materials relies on physics-based object representations: Behavioral and computational evidence

Bi, Wenyan; Shah, Aalap D; Wong, Kimberly W.; Scholl, Brian J.; Yildirim, Ilker

doi:10.1167/jov.21.9.2624

Cited by 2 publications

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…5A). These data are consistent with findings that probabilistic physical simulations can account for behavioral judgments on single scenarios that resemble ours [10,51,7,13]. However, our results go beyond prior work in several ways.…”

Section: Resultssupporting

confidence: 92%

“…We have therefore designed and used our benchmark to assess a wide range of approaches in physics learning and prediction, sampling representatives from several model types: vision models that make pixel-level future predictions via fully convolutional architectures, [23,1,36,21,35,70,40,41,66,30,34,54,27]; those that either explicitly learn object-centric representations of scenes [64,33,19,27,50] or are encouraged to learn about objects via supervised training [56,62]; and physics dynamics models that operate on objector particle-graph representations provided as input [16,9,37,8,61,52,11,42,2,57,69,47]. Models that perform physical simulation on a graph-like latent state are especially attractive candidates for making human-like predictions, because non-machine learning algorithms that simply add noise to a hard-coded simulator have been found to accurately capture human judgments about a variety of physical scenarios [10,51,7,13]. Consistent with this, recurrent graph neural networks supervised on physical simulator states make extremely accurate predictions about full object trajectories [42,37,38,53].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Physion: Evaluating Physical Prediction from Vision in Humans and Machines

Bear¹,

Wang²,

Mrowca³

et al. 2021

Preprint

View full text Add to dashboard Cite

While machine learning algorithms excel at many challenging visual tasks, it is unclear that they can make predictions about commonplace real world physical events. Here, we present a visual and physical prediction benchmark that precisely measures this capability. In realistically simulating a wide variety of physical phenomena -rigid and soft-body collisions, stable multi-object configurations, rolling and sliding, projectile motion -our dataset presents a more comprehensive challenge than existing benchmarks. Moreover, we have collected human responses for our stimuli so that model predictions can be directly compared to human judgments. We compare an array of algorithms -varying in their architecture, learning objective, input-output structure, and training data -on their ability to make diverse physical predictions. We find that graph neural networks with access to the physical state best capture human behavior, whereas among models that receive only visual input, those with object-centric representations or pretraining do best but fall far short of human accuracy. This suggests that extracting physically meaningful representations of scenes is the main bottleneck to achieving human-like visual prediction. We thus demonstrate how our benchmark can identify areas for improvement and measure progress on this key aspect of physical understanding. * / * * Equal contribution Preprint. Under review.

show abstract

Section: Resultssupporting

confidence: 92%

Section: Related Workmentioning

confidence: 99%

Physion: Evaluating Physical Prediction from Vision in Humans and Machines

Bear¹,

Wang²,

Mrowca³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…1a and 1b). And indeed, recent computational work confirms that apprehending such relationships requires simulation of physical principles, beyond brute image metrics (Bi et al, 2021). Accordingly, we will refer to such phenomena as "cloth physics.…”

Section: "Cloth Physics"mentioning

confidence: 97%

Seeing Soft Materials Draped Over Objects: A Case Study of Intuitive Physics in Perception, Attention, and Memory

Wong

Soltani

et al. 2022

Psychol Sci

Self Cite

View full text Add to dashboard Cite

We typically think of intuitive physics in terms of high-level cognition, but might aspects of physics also be extracted during lower-level visual processing? Might we not only think about physics, but also see it? We explored this using multiple tasks in online adult samples with objects covered by soft materials—as when you see a chair with a blanket draped over it—where you must account for the physical interactions between cloth, gravity, and object. In multiple change-detection experiments ( n = 200), observers from an online testing marketplace were better at detecting image changes involving underlying object structure versus those involving only the superficial folds of cloths—even when the latter were more extreme along several dimensions. And in probe-comparison experiments ( n = 100), performance was worse when both probes (vs. only one) appeared on image regions reflective of underlying object structure (equating visual properties). This work collectively shows how vision uses intuitive physics to recover the deeper underlying structure of scenes.

show abstract

Perception of soft materials relies on physics-based object representations: Behavioral and computational evidence

Cited by 2 publications

References 0 publications

Physion: Evaluating Physical Prediction from Vision in Humans and Machines

Physion: Evaluating Physical Prediction from Vision in Humans and Machines

Seeing Soft Materials Draped Over Objects: A Case Study of Intuitive Physics in Perception, Attention, and Memory

Contact Info

Product

Resources

About