Unsupervised learning predicts human perception and misperception of gloss

Abstract

1 Summary Gloss perception is a challenging visual inference that requires disentangling the contributions of reflectance, lighting, and shape to the retinal image [1][2][3]. Learning to see gloss must somehow proceed without labelled training data as no other sensory signals can provide the 'ground truth' required for supervised learning [4][5][6]. We reasoned that paradoxically, we may learn to infer distal scene properties, like gloss, by learning to compress and predict spatial structure in proximal image data. We hypothesised that such unsupervised learning might explain both successes and failures of human gloss perception, where classical 'inverse optics' cannot. To test this, we trained unsupervised neural networks to model the pixel statistics of renderings of glossy surfaces and compared the resulting representations with human gloss judgments.The trained networks spontaneously cluster images according to underlying scene properties such as specular reflectance, shape and illumination, despite receiving no explicit information about them. More importantly, we find that linearly decoding specular reflectance from the model's internal code predicts human perception and misperception of glossiness on an image-by-image basis better than the true physical reflectance does, better than supervised networks explicitly trained to estimate specular reflectance, and better than alternative image statistic and dimensionality-reduction models. Indeed, the unsupervised networks correctly predict well-known illusions of gloss perception caused by interactions between surface relief and lighting [7,8] which the supervised models totally fail to predict. Our findings suggest that unsupervised learning may explain otherwise inexplicable errors in surface perception, with broader implications for how biological brains learn to see the outside world.2 Highlights  We trained unsupervised neural networks to synthesise images of glossy surfaces  They spontaneously learned to encode gloss, lighting and other scene factors  The networks correctly predict both errors and successes of human gloss perception  The findings provide new insights into how the brain likely learns to see 3 Results and DiscussionThe central intuition behind our findings is that learning to compress the complex image structure created by reflections from glossy surfaces into a highly compact code forces the brain to discover representations that partially-but imperfectly-disentangle the distal physical factors responsible for variations within and between images. This potentially explains both the broad successes and specific pattern of errors known to occur in gloss perception [2,3,[7][8][9][10][11][12]. To test this, we rendered 10,000 images from a virtual world of bumpy frontoparallel surfaces with either high or low specular reflectance, random colour and depth of surface relief, illuminated by six natural light fields (Figure 1A-B). We trained ten instances of an unsupervised PixelVAE network [13,14] on this dataset. The model's training objecti...

An ecologically motivated image dataset for deep learning yields better models of human vision

“…We here trained all DNNs to optimize for categorization performance. While this task is undoubtedly of ecological relevance, the explanatory power of unsupervised objectives ( 35 – 37 ), semantically better-informed training targets, and their interplay with ecoset will be worth considering going forward.…”

Diverse deep neural networks all predict human IT well, after training and fitting

“…Recurrent networks can recycle neural resources to flexibly trade speed for accuracy in visual recognition and show great promise as models of temporal dynamics in visual cortex (Güçlü & van Gerven, 2017;Kietzmann et al, 2019;Nayebi et al, 2018;Spoerer et al, 2017;van Bergen & Kriegeskorte, 2020). Unsupervised learning objectives provide rich and ecologically feasible ways of getting complex knowledge about the visual world into the brain (Fleming & Storrs, 2019;Storrs & Fleming, 2020). Models should be able to predict both internal representations and behavioural data (Funke et al, 2020;Jozwik et al, 2017;), and will be tested using larger datasets, with higher noise ceilings, and with stimuli designed to tease apart the differences between model predictions (Golan et al, 2019).…”

Visual perception of liquids: Insights from deep neural networks

“…Artificial neural networks have changed how we model visual and neural processes. Feedforward neural networks-despite lacking top-down and lateral processing [46,47]-have proven to be an insightful tool for vision science [48][49][50][51][52][53][54][55][56] and currently provide the most successful computer vision models as well. Many processing aspects of the human visual system are not properly simulated by feedforward designs and the sensory input methods and learning regimen leave much to be desired as well [57][58][59][60][61][62].…”